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Implementation,  there  Is  a  need  for  government  Environmental  -Hardware 
engineers  and  CERT  facilities  and  MIL-STD-810  and  MIL-STD-781  environmental 
criteria  should  complement  each  other. 


FOREWORD 


This  Report  was  prepared  for  the  PRAM  Program  Office,  Aeronautical  System 
Division  (AFSC),  Wrl ght-Patterson  AFB,  to  record  the  proceedings  of  the  OoD 
Combined  Environment  Reliability  Test  (CERT)  Workshop  by  The  Analytic 
Sciences  Corporation  and  Universal  Technology  Corporation  under  Contract 
F33657-80-C-0255.  The  proceedings  document  contains  summaries  or  written  ver¬ 
sions  of  presentations  and  working  group  discussions.  The  Workshop  findings. 
Issues  and  recommendations  are  summarized  In  an  Executive  Sumnary  at  the 
beginning  of  the  Report. 
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EXECUTIVE  SUMMARY 


The  DoO  CERT  Workshop  was  held  2-4  June  1981  at  the  Presidential  Hotel  in 
Atlanta,  Georgia.  Seventy-six  representatives  from  Industry  and  government 
attended  the  workshop.  The  purpose  was  to  develop  recommendations  concerning 
the  use  of  CERT  In  the  acquisition  process.  A  summary  of  the  Issues  and 
recommendations  Is  presented  below.  Section  IV  of  this  document  contains  a 
more  detailed  description  of  the  discussion,  issues  and  recommendations  of  the 
working  groups. 

A.  CERT  MANAGEMENT/COST  EFFECTIVENESS/DOD  5000.40  WORKING  GROUP 


Issue  #1: 

Educate  the  aerospace  community  on  the  uses  of  CERT  and  results  of  the 
workshop. 

Recommendation;  In  addition  to  the  proceedings,  a  presentation  should  be  pre¬ 
pared  and  presented  to  appropriate  organizations,  such  as  HQ  AF  Systems  Comma nc 
and  Joint  Logistics  Commands.  Also,  presentations  can  be  made  to  different 
engineering  conferences  and  symposia  as  well. 

Issue  #2: 

Identify  the  most  effective  means  of  Implementing  the  use  of  combined  environ¬ 
ments  testing  In  acquisition  and  modification  programs  throughout  the  DoD. 

• 

Recommendation:  A  dual  approach  was  recommended:  top-down  and  bottom-up. 

It  was  recommended  that  DoD  issue  a  letter  to  each  of  the  three  services 
requesting  Identification  of  a  focal  point.  Also,  provide  data  from  programs 
successfully  Implementing  the  combined  environments  testing  concepts  (l.e. 
Common  Strategic  Doppler,  Sparrow  Missile,  Omega)  be  consolidated  and  publi¬ 
cized. 


Issue  #3: 


DoD  Directive  5000.40  Implementation  Experience. 

Recommendation:  OoO  Directive  5000.40  was  formally  issued  eleven  months  ago. 
Since  then,  DoD  is  considering  revisions  to  the  entire  system  acquisition  pro¬ 
cess.  Therefore,  no  recommendations  were  made. 

Issue  #4: 

Meed  for  Environmental  Hardware  Engineer. 

Recommendation:  The  consensus  was  that  this  technical  discipline  would  be 
greatly  beneficial  to  a  System  Program  Office  in  developing  test  criteria. 

Issue  #5: 

Identification  of  CERT  Benefits. 

Recommendation:  Combined  environment  testing  is  an  improved  approach  to 
testing.  By  combining  environmental  stresses,  this  would  lead  to  a  reduction  in 
the  number  of  tests  and  equipment  items  required  tor  testing.  Also,  by  more 
closely  simulating  operational  environments,  a  better  measure  of  field  reliabi¬ 
lity  can  be  obtained.  (See  Section  IV,  Figure  3) 

.1.  TECHNICAL  APPLICATIONS  WORKING  GROUP 

Issue  #1: 

The  potential  uses  of  CERT. 

Recommendation:  CERT  should  be  based  upon  thorough,  realistic  engineering 
analysis.  It  is  applicable  in  any  situation  in  which  it  can  improve  effec¬ 
tiveness  or  efficiency.  Test  criteria  should  be  developed  for  each  test  and 
application,  and  should  not  be  applied  as  a  blanket  methodology. 


Issue  #2: 


Use  of  CERT  In  lieu  of,  or  conjunction  with,  flight  testing. 

Recommendation:  While  CERT  should  not  be  considered  as  a  complete  replace¬ 
ment  for  flight  testing.  It  can  be  used  as  a  pre-flight  test  design  and  screen. 
It  was  also  suggested  that  CERT  should  be  used  at  as  great  a  level  of  assembly 
as  feasible. 

Issue  #3: 

Use  of  CERT  for  multi-application  equipment. 

Recommendation:  The  most  severe  environment,  Including  the  life  cycles  of  a 
multiple  platform  or  mission,  should  be  used  as  a  baseline  In  establishing  a 
test  environment  on  design  Improvement. 

Issue  #4: 

Differences,  if  any,  between  reliability  tests  for  operational  and  compliance 
characterization. 

Recommendation:  Representative  environmental  test  profiles  should  use  realistic 
stresses.  User's  needs  should  determine  which  test  profiles  are  considered 
represented ve  for  operational  versus  compliance  characterization. 

Issue  #5: 

Use  of  CERT  as  a  substitute  for,  or  In  conjunction  with,  other  tests. 

Recommendation:  CERT  should  be  used  as  a  method  to  replace  test  time  and  costs 
In  terms  of  combining  test  objectives,  such  as  environmental  qualification, 
reliability  Improvement,  etc. 


C.  FACILITIES  WORKING  GROUP 


I stu«  #1: 

Availability  of  a  government-owned  and  operated  CERT  facility. 


Reco— endatlon:  It  is  necoiended  that  both  industry  and  government  have  CERT 
facilities.  Government  facilities  should  be  used  to  develop  test  techniques 
and  specifications,  assure  Impartial  evaluation  of  competing  contractors,  and 
assess  product  reliability.  Contractor  facilities  should  be  used  to  design, 
evaluate,  and  correct  deficiencies  of  systems,  especially  in  early 
development. 


D.  MIL-STD-781/APPEN0IX  B  AND  MIL-STD-810  WORKING  GROUP 


Issue  #1 : 

Differences  between  the  two  standards. 

Reconmendatlon:  MIL-STD-810  Is  used  as  a  high  level  test  criteria  to  deter¬ 
mine  whether  equipment  will  function  and  withstand  severe  environmental  con¬ 
ditions  for  particular  positions  in  the  life  cycle  profile.  It  is  a  short 
duration  test.  MIL -STD-781 /Appendix  B  should  be  used  to  represent  those  levels 
which  are  found  for  the  majority  of  the  life  cycle  to  obtain  reliability  (i.e. 
MTBF)  Information.  It  is  a  long  duration  test. 

Issue  #2: 

Environmental  Oata  Base. 

spcommendatlon'  A  focal  point  should  be  established  to  maintain  a  readily 
available  document  which  contains  a  common  environmental  data  base  of  existing 

olatforms. 


ix 


Issue  #3: 


Establish  minimum  vibration  level  for  retaining  test  effectiveness. 

Recommendation;  Delete  the  requirement  to  vibrate  during  the  test  time  when  the 
mission  profile  vibration  level  is  equal  to  or  less  than  0.001  g2/Hz. 


Issue  #4: 

Research  and  development  plan  for  MIL-STD-781  and  MIL-STD-810. 

Recommendation;  Establish  a  five-year  research  and  development  plan  to  obtain 
and  evaluate  Improved  methods,  procedures  and  techniques. 
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SECTION  I  -  OVERVIEW 


1.0 


WORKSHOP  INTRODUCTION 


A.  Purpose  \ 

v 

The  purpose  of  these  proceedings  is  to  document  the  results  of  the  DoD 
CERT  Workshop  conducted  on  2-4  June  1981  in  Atlanta,  Georgia.  The  DoD  CERT 
Workshop  was  held  to  develop  recommendations  concerning  the  use  of 
CERT  in  the  acquisition  process.  These  recommendations  covered  the  cost- 
effectiveness,  applicability  and  benefits  of  CERT. 

B.  Scope 

A  three-day  workshop  was  conducted  to  address  the  Issues  and  applica¬ 
tions  of  CERT  testing.  An  outline  of  the  Workshop  Agenda  Is  Included  on 
pages  1.1.3  and  1.1.4.  The  keynote  address,  given  by  Brigadier  General  Elbert 
Harbour,  stated  the  purpose  of  the  workshop.  Key  members  of  govern¬ 
ment  and  industry  participated  In  a  distinguished  panel  forum  which  gave  a 
management's  perspective  In  the  area  of  acquiring  reliable  weapon  systems 
and  the  effect  which  reliability  testing  can  have  on  these  systems.  Several 
technical  papers  were  given  which  presented  the  many  different  viewpoints 
and  assumptions  which  have  evolved  from  the  use  of  CERT.  This  was  done 
to  achieve  a  corrmon  ground  for  the  participants. 

The  workshop  discussion  sessions  consisted  of  four  working  groups:  CERT 
Management/Cost  Effectiveness;  Technical  Applications;  Facilities;  and,  Mli-STD- 
781  and  MIL-STD-810.  Each  group  was  given  specific  issues  to  address.  The  final  outout 
of  the  groups  were  recommendations  which  will  impact  DoD  policy  on  combined 
environment  testing. 

C.  Background 

Over  the  past  six  years,  there  has  been  an  Intensive  effort  to  Improve 
the  nature  and  quality  of  environmental  and  reliability  testing.  Many  of 


these  testing  methods  have  been  labeled  "CERT11.  This  has  led  to  the  acronym 
CERT  becoming  a  generic  term  for  any  kind  of  environmentally-based  testing  In 
which  realistic  usage  environments  are  combined  In  a  practical  manner  and  varied 
as  a  function  of  time.  The  growing  universal  usage  of  this  acronym  has  led  to 
many  misconceptions  as  to  what  Is  practical  and  cost-effective  In  any  given 
situation. 

The  DoD  CERT  Worshop  was  conceived  to  focus  on  the  Identification  of  vrfiat 
has  beery  found  to  be  practical,  useful  and  cost-effective.  Results  from  exten¬ 
sive  government  and  industry  experience  in  the  use  of  CERT  would  be  presented. 
These  experiences  would  be  used  to  develop  recommendations  from  the  workshop 
which  could  serve  as  a  guide  to  future  acquisitions.  The  output  of  this 
workshop  is  to  be  given  the  widest  possible  distribution  to  policy  makers,  pro¬ 
ject  engineers  and  test  practitioners  to  serve  as  a  consolidated  assessment  of 
where,  when  and  who  should  be  doing  CERT. 


1.1.2 


CERT  WORKSHOP  AGENDA 


Monday,  1  June  1981 

1800-2100  Pre-Registration 
Tuesday,  2  June  1981 


0800-0900 

Final  Registration 

0900-0930 

Welcoming  Address 

0930-1000 

Distinguished  Panel  Forum 

1000-1015 

Refreshment  Break 

1015-1115 

Distinguished  Panel  Forum  (continued) 

1115-1130 

Question/Answer  Period 

1130-1300 

Group  Luncheon 

1300-1500 

Selected  Technical  Papers  (4  speakers) 

1500-1520 

Refreshment  Break 

1520-1700 

Selected  Technical  Papers  (3  speakers) 

1700 

Adjourn 

1.1.3 


Wednesday,  3  June  1981 

0830-0900  Assignments/Instructions  for  Working  Group  Sessions 


0900-1200 

Working  Group  Sessions 

1.  CERT  Management/Cost  Effectlveness/DOD  5000.40 

2.  Technical  Applications  -  Advances  In  Techniques 

3.  Facilities  (In-House  vs  Industry)  -  Current 

Capabilities 

4.  MIL-STD-781 /Appendix  B  A  MIL-STD-810  Proposals 

1200-1330 

Group  Luncheon 

1330-1700 

Working  Group  Sessions  (continued) 

1700 

Adjourn 

Thursday,  4  June  1981 

0830-1000 

Wrap-up  of  Working  Group  Sessions/Reports 

1000-1015 

Refreshment  Break 

1015-1115 

General  Session  on  Working  Group  Reports  (15  min.  sum¬ 
mary  of  each) 

1115-1130 

Closing  Remarks 

1130 

Adjourn 

1.1.4 


CERT 


COMBINED  ENVIRONMENT  RELIABILITY  TESTING 


Brigadier  General  Elbert  E.  Harbour 
Deputy  for  Airlift  A  Trainer  Systems 
Aeronautical  Systems  Division 
Wright-Patterson  AFB,  Ohio 


It  Is  ny  pleasure  to  be  given  the  opportunity  to  say  a  few  words  about 
Combined  Environment  Reliability  Testing  (CERT).  During  the  next  few 
minutes  I  would  like  to  concentrate  on  three  Issues.  First,  “Is  reliabi¬ 
lity  of  a  weapon  system  Important?"  Secondly,  "How  much  reliability  should 
a  weapon  system  have?"  Third,  "Can  reliability  be  predicted  accurately?" 

To  begin,  let's  look  at  a  little  history  regarding  the  weapon  system 
called  the  flying  machine.  Contrary  to  popular  opinion,  the  government's 
first  contract  to  buy  a  flying  machine  was  not  with  Orville  and  Wilbur 
Wright.  Well  before  the  Wright's  first  flight,  the  War  Department's  Board 
of  Ordnance  and  Fortification  secretly  allocated  $50,000  to  Dr.  Samuel  P. 
Langley,  who  was  subsequently  unable  to  produce  a  promised  flying  machine. 
When  this  Information  became  generally  known,  both  Congress  and  the  Press 
had  been  extremely  critical  of  this  so-called  wastage  of  public  funds.  As 
a  result,  the  Board  of  Ordnance  and  Fortification  declined  to  enter  Into 
negotiations  with  the  Wright  brothers  until  a  machine  was  produced,  which,  by 
actual  operation  was  shown  to  be  able  to  produce  horizontal  flight  and  to 
carry  an  operator. 

However,  this  fly-before-buy  program  with  the  Wright  Brothers,  had  by 
today's  standard  a  serious  shortcoming  In  that  there  were  no  CERT  or 
reliability  demonstration  requirements,  only  performance  requirements.  The 
airplane  development  program  was  apparently  run  under  the  belief  that  there 
is  a  time  and  place  for  everything  and  the  Wright  flying  machine  was  not 
the  time  nor  place  for  CERT.  If  that  Is  the  case,  the  question  Is  then, 
"should  there  have  been?" 


The  answer  to  this  question  is  rather  obvious,  especially  If  you  want 
to  remove  the  uncertainty  about  our  capability  to  successfully  conduct 
military  actions.  We  must  know  the  reliability  of  our  weapon  systems;  this 
in  turn  drives  force  size.  Take  for  example  a  cannon  equipped  aircraft, 
say  similar  to  the  A-10.  If  this  aircraft  had  a  kill  accuracy  of  .2  and  a 
reliability  factor  of  .2,  together  the  cannon  equipped  aircraft  would  have 
a  kill  probability  of  .04.  Hence,  in  order  to  assume  a  50  percent  per  day 
kill  on  1,000  targets,  we  would  need  12,500  aircraft  sorties  per  day.  In 
the  above  example,  there  is  a  strong  indication  that  both  performance 
(accuracy)  and  reliability  are  candidates  for  improvement.  The  above 
example  makes  a  clear  case  for  reliability.  But  how  much  is  the  question. 

The  answer  to  the  second  question  on  how  much  reliability  is  far  more 
complex.  Perhaps  there  are  no  standard  answers.  Another  example.  Let's 
suppose  that  $70  billion  up-front  R4D  cost  will  drive  reliability  of  our 
cannon  equipped  weapon  system  up  to  the  point  that  the  lowest  life  cycle 
cost  is  achieved.  Is  $70  billion  affordable?  Seventy  billion  dollars  will 
consume  the  entire  Air  Force  budget.  How  can  we  pay,  feed  and  train  the 
crews?  This  is  about  the  point  where  some  would  lose  sight  of  the  objec¬ 
tive  and  raise  the  question  "Is  the  cannon  equipped  aircraft  affordable?", 
but  is  a  separate  issue.  What  we  really  do  is  make  a  subjective  tradeoff. 
We  must  look  at  reliability  projected  cost  curves  and  pick  a  point,  a 
reasonable  point. 

Now  for  the  third  issue,  "Can  reliability  really  be  predicted?"  There 
are  those  experts  who  say  that  laboratory  reliability  demonstrations  can 
predict  reliability  and  there  are  those  experts  that  say  the  laboratory 
cannot  simulate  field  conditions.  I  have  also  been  advised  by  reliability 
experts  that  there  is  a  definite  correlation  between  the  laboratory 
reliability  demonstration  and  field  experience,  but  that  this  relationship 
varies  from  system  to  system  and  they,  the  experts,  don't  know  how  to  pre¬ 
dict  that  relationship. 

I  don't  intend  to  get  embroiled  in  that  issue  but  I  am  sure  that  we  do 
know  how  to  design-in  reliability,  how  to  predict  and  measure  stress 


points  during  Initial  design  of  the  system  and  how  to  continue  this  method 
throughout  development  of  a  system.  So  rather  than  argue  about  how  many 
teeth  a  horse  can  have,  like  the  earlier  philosopher,  let's  just  count  the 
horse's  teeth  and  get  on  with  CERT. 

But  CERT,  I  believe,  must  be  subjected  to  the  same  scrutiny  and  logic 
In  arriving  at  how  much  Is  enough,  which  we  currently  do  In  determining  how 
much  reliability  Is  enough.  Theoretically,  we  could  build  a  statistically 
significant  number  of  our  cannon-equipped  weapon  systems  and  conduct  CERT 
on  them  until  the  desired  reliability  was  achieved.  At  this  point,  produc¬ 
tion  could  begin  with  confidence.  But,  this  Isn't  practical. 

On  the  other  hand,  some  environment  and  reliability  testing  Is  essen¬ 
tial.  This  reminds  me  of  the  story  of  an  old  friend  of  mine,  fy  friend  Is 
a  fanner  and  he  was  approached  by  a  city  slicker  one  day.  The  city  slicker 
wanted  to  buy  the  farmer's  old  plow  horse.  Now  this  horse  was  not  In  the 
best  of  health  —  and  the  farmer  knew  It.  But  the  city  slicker  wanted  a 
horse  and  offered  the  farmer  $100.  The  farmer  really  tried  to  caution  the 
city  slicker  on  the  horse's  health,  but  the  city  slicker's  mind  was  made 
up.  The  $100  changed  hands  and  the  city  slicker  had  himself  a  horse.  You 
will  notice  that  no  CERT  was  performed,  not  even  a  ride- before- buy. 

A  couple  of  weeks  passed  and  the  farmer  saw  the  city  slicker  In  town. 
The  city  slicker  told  him  that  the  horse  had  died  Just  after  he  got  It 
home.  The  farmer  said  he  was  sorry  but  that  there  was  no  way  he  could  give 
back  the  money  -  he  had  already  spent  It.  The  city  slicker  stated  that  he 
didn't  want  his  money  back;  In  fact,  he  said  he  had  made  $1000  off  of  the 
horse.  The  farmer  asked,  "How  In  the  world  did  you  make  $1000  off  of  a 
dead  horse?"  The  city  slicker  said  that  he  had  a  raffle  and  sold  1000 
tickets  for  $1  apiece.  The  1000  people  must  have  really  been  upset,  the 
farmer  said,  when  they  found  out  that  the  horse  was  dead.  No,  said  the 
city  slicker,  only  one  —  and  I  gave  him  Ms  dollar  back!  Unlike  the  city 
slicker,  we  can't  get  our  dollars  back  If  we  buy  a  dead  horse.  While  CERT 
won't  keep  us  from  the  purchase.  It  certainly  will  provide  Insight  as  to  our 
horse's  longevity. 


1.2.3 


CERT  Is  Important  and  Is  a  must  vrtien  the  performance  technology  Is  In 
hand.  The  C-X  Is  a  prime  example.  THe  C-X  techno  jgy  is  not  only  in  hand 
but  has  been  demonstrated  by  the  commercial  fleet.  When  we  started  the  C-X 
we  did  not  call  our  approach  CERT,  but  rather  environmental  stress  screen¬ 
ing  (ESS).  ESS  testing  is  designed  to  stimulate  failures  by  applying 
random  vibration  and  temperature  cycling  to  all  of  the  electronic  equip¬ 
ment.  The  purpose  of  the  screening  is  to  identify  weak  parts,  workmanship 
defects  and  other  anomalies  and  to  remove  them  from  the  equipment  prior  to 
building  the  subsystem,  and  finally  the  system. 

In  closing,  let  me  caution  you.  Although  CERT  is  an  approach  wnose 
time  has  come,  it  must  be  instituted  with  sound  judgment.  Keep  the  objec¬ 
tive  clearly  In  mind  concerning  what  is  practical  and  cost  effective  in 
your  own  particular  situation;  and  lastly,  do  not  become  a  cult  that  has 
lost  sight  of  the  objective,  If  you  lose  sight  of  the  objective,  doubling 
the  effort  is  futile.  A  comprehensive  reliability  program  must  be 
established  and  conducted  which  will  satisfy  the  reliability  requirements 
contained  in  the  specifications.  The  bottom  line  is  to  design  for  reliabi¬ 
lity,  but  work  with  the  developers  and  prove  that  you  have  a  better 
mousetrap  and  the  developers  will  beat  a  path  to  your  door.  Finally, 
remember  If  you  do  try  to  force  CERT  upon  the  developer's  world,  it  will 
not  work,  because  there  are  more  of  the  developers  than  there  are  of  you. 
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SECTION  II  -  DISTINGUISHED  PANEL  FORUM 


INTRODUCTION 


The  purpose  of  the  Distinguished  Panel  Forum  was  to  bring  together 
some  of  the  key  leaders  of  government  and  Industry  to  present  management's 
perspective  on  the  affects  of  reliability  testing.  Each  panel  member  was 
Invited  based  on  his  widely  recognized  expertise  and  experience  In  the 
area  of  acquiring  reliable  weapon  systems.  The  panel  membership  consisted 
of  the  following  Individuals: 


Panel  Moderator: 

Colonel  Robert  Loplna 
Deputy  for  Engineering 
Aeronautical  Systems  Division 


Panel  Members/Speakers: 


Colonel  Thomas  Musson 
Asst  for  Reliability  and  Maintainability 
Office  of  Under-Secretary  of  Defense 
for  Research  and  Engineering 


Mr.  Robert  Brown 
Asst  to  the  Commander 
Hq  Acquisition  Logistics 
Division 


Mr.  Robert  Hancock 
Manager  of  Combined  Environments 
Technology  and  Test  Organization 
Vought  Corporation 
1979-80  National  Institute  of 
Environmental  Sciences  President 


Mr.  Jack  Lavery 

Asst  for  Product  Assurance 

Hq  Air  Force  Systems  Command 


Summaries  of  the  presentations  by  these  four  speakers  are  Included  In 
the  following  pages. 


2.1 


SUWARY  OF  PRESENTATION 
by 

Robert  V.  Brown 
Assistant  to  the  Commander 
Air  Force  Acquisition  Logistics  Division 
Wright-Patterson  AFB,  Ohio 


Mr.  Brown  began  by  defining  the  AFALD  role  as  one  of  ombudsman  for 
the  maintalners  and  users  of  systems  and  equipment.  He  indicated  that 
more  accurate  methods  of  predicting  field  reliability  are  sorely  needed. 

In  the  absence  of  such  better  methods,  his  standard  guidance  within  the 
AFALD  for  logistics  planners  has  been  that  the  development  community's 
reliability  predictions  should  be  lowered  by  factors  as  high  as  10. 

Brown  indicated  that  performance  over  time  In  a  statistically  signifi¬ 
cant  sense  differentiates  CERT  from  CET,  and  that  the  "fix"  in  TAF 
(Test-Analyze-FIx)  really  means  "fix  the  design"  or  "redesign",  as 
contrasted  to  simply  restoring  a  failed  unit  to  operational  status.  He 
emphasized  that  fixing  the  design  (redesigning  as  required)  is  one  of  the 
most  Important  concerns  of  today's  logisticians.  CERT  appears  very  close 
to  being  ready  for  implementation,  and  he  urged  the  workshop  to  Implement 
the  concept  expeditiously.  Finally,  Mr.  Brown  closed  with  several  specific 
challenges  to  the  Workshop: 

1.  Determine  how  to  measure  the  operational  benefit  of  reliability. 

2.  Consider  the  benefits  of  faster  redesign  realized  through 

locating  the  CERT  facilities  at  contractor’s  plants. 

3.  The  Military  Standards  are  guides.  Are  they  sufficient  in  clarity 

and  scope? 

4.  Find  the  best,  way  to  implement  the  DoD  policy  on  Reliability  Testing. 


SUMMARY  OF  PRESENTATION 
by 

Jack  Lavery 

Assistant  for  Product  Assurance 
Hq  Air  Force  Systems  Command 
Andrews  AFB,  Maryland 


Mr.  Lavery  emphasized  that  it  was  time  to  bring  CERT  Into  operational 
use.  CERT  should  become  Institutionalized.  Mr.  Lavery  posed  a  number  of 
challenges  for  the  workshop  to  address  and  come  up  with  guidance  for  senior 
management.  He  emphasized  objectivity  and  earning  the  credibility  of  mana¬ 
gement.  He  recommended  a  balanced  program  with  tallorabil ity  to  specific 
requirements.  He  further  emphasized  the  need  for  clear  communications  with 
management  showing  the  return  on  investment. 

It  was  pointed  out  that  there  are  Important  programs  today  which 
needed  CERT,  such  as  the  B-l  bomber  and  the  Air  Launched  Cruise  Missile 
(ALCM) .  He  recommended  that  the  Working  Groups  come  up  with  guidance  for 
tailoring  CERT  In  the  acquisition  process.  He  suggested  that  we  establish 
the  relationship  between  MIL-STD-781C  or  MIL-STD-810C  and  the  acquisi¬ 
tion  process. 

Mr.  Lavery  commented  that  it  was  often  easier  to  define  an  environmen¬ 
tal  test  for  a  piece  of  avionics  than  to  Identify  the  mission  profile  that 
the  avionics  would  actually  see  during  Its  life-cycle.  A  chart  was  shown 
(next  page)  which  compared  in-field  experience  with  test  data  for  the  F-15 
avionics. 

Mr.  Lavery  charged  the  Workshop  attendees  with  coming  up  with  a 
planned  course  of  action  to  institutionalize  CERT. 


2.3 


F-15  AVIONICS  AVAILABILITY  STUDY 


AFR  68-1  (1977)  McDONNELL  (1979) 


INFLIGHT  FAILURE 


INDICATORS 

100 

100 

"CND" 

47 

43 

REPAIR  ON  EQP 

3 

21 

REMOVALS 

60 

36 

"RTOK” 

12 

11 

SUMMARY  OF  PRESENTATION 
by 

Robert  Hancock 

Manager,  Combined  Environments  Technology 
and  Test  Organization 
Vought  Corporation 
Dallas,  Texas 


Mr.  Hancock  summarized  for  the  participants  of  the  DoD  CERT 
Workshop,  the  position  of  the  Institute  of  Environmental  Sciences  (IES) 
with  regards  to  CERT.  There  is  a  need  to  identify  CERT  in  terms  of  quali¬ 
fiers  such  as  platform  (aircraft,  missile,  shipboard . ),  environments 

(temperature,  sinusoidal  vibration,  random  vibration,  ...),  test  type 
(improvement,  reliability  growth,  estimate  reliability,...),  and  hardware  level 
(system,  subsystem,  component,  ...). 

IES  has  been  assessing  the  CERT  facility  requirements  as  a  result  of 
direction  of  MIL-ST0-781C  and  DoD  Directive  5000.40.  Their  estimate  of 
required  expenditure  of  $400  million  over  10  years  (from  1975  to  1985)  Is 
currently  at  $100  million  after  5  years,  thus  at  the  half-way  point 
only  25%  of  the  necessary  funds  to  facilitate  have  been  expended.  The  con¬ 
sensus  is  that  Industry  will  put  CERT  facilities  in  place  when  there  Is 
justification.  The^e  is  a  need  for  a  "new  look”  CERT  Program  Status  which 
would  define  the  role  of  CERT  in  the  procurement  cycle  and  would  be 
advanced  to  upper  management,  perhaps  through  MIL-STD-XXX. 

IES  would  recommend  that  the  Government: 

-  Require  CERT  in  validation  phase 

-  Continue  CERT  in  the  development  phase 

-  Specify  CERT  In  RFP 

-  Adopt  "new  look"  document  (MIL-STD-XXX). 

Copies  of  Mr.  Hancock’s  viewgraphs  are  Included  in  the  following 
pages. 
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IES  CONCENSUS  ON  CERT 


TEST  METHODS 

•  NEED  CLARIFICATION  OF  CERT  VS.  CET  (NO  RELIABILITY  OUTPUT) 

•  CERT  W/L-C  PROFILING  FOR  RELIABILITY  GROWTH  (TAAF),  RELIABILITY  DEMO 
AND  RATE 

•  CET  W/L-CP  HAS  APPLICATIONS  FOR  DESIGN  PROOF  TE8TING 

•  CET  (WITHOUT  PROFILES)  FOR  EARLY  DESIGN  STUDIES,  DESIGN  EVAL  AND  PROD. 
SCREENING 

LIFE-CYCLE  ENVIRONMENTS 

•  MAJOR  EFFORT  NEEDED  TO  COMPILE  TRACEABLE  DATA  BANK 

•  CAN  SPECIFY  TESTS  MUCH  MORE  ACCURATELY  THAN  DEFINE  ENVIRONMENTS 

FACILITIES 

•  WILL  PUT  IN  PLACE  WHEN  PROCUREMENT  PRACTICES  ALLOW  JUSTIFICATION 

•  INITIAL  1S76  ESTIMATE  FOR  781c  DEMO  -  130  MIL/5  YRS. 

•  ESTIMATES  REVI8ED  TO  SATISFY  TAAF  AND  CET  SCOPE  -  400  MIL/ 10  YRS. 

•  SPENT  APPROX.  1100  M  BY  6-80,  V*  OF  NEEDED  RATE  FOR  1886  IOC. 


NEW-LOOK/CERT  OVERALL  PROGRAM''  STATUS 


CONCEPT  CEFINITION 


VALIDATION 


ENQR 

DEVELOPMENT 


I  _ _  PILOT  . 

]  USE  ^"T 
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NEW-LOOK  AND  CERT  RISK  FACTORS 
(VIEWED  AT  TIME  OF  FSD  PROPOSAL) 


OLD-LOOK 


1  -  2  HR  MTSF 
INCRCASINQ 

K:t 

MCTHOOS  IN  PLACE 
MSTHOOS  IN  PLACE 
KNOWN 
HE  PLACE 
KNOWN 

•PARCS,  ECPS 
OCPRtmVE 
BUSHWSS  AS  USUAL 


CHARACTERISTIC 

-  SYSTEMS  RELIABILITY  - 

-  OPS.  SUPPORT  COSTS  - 

-  LAS.  REL.  MEAS.  DISPARITY  - 

-  RELIABILITY  PRSDICTtONS  - 

-  L-C  COSTtNO  - 

-  TEST  ID  DC  SION)  REOMTS  - 

-  TEST  EQUIPMEirr  - 
-  LAS  TCSTINO  COSTS  - 


NEW-LOOK 

•  -  •  HR  MTBP  (Mil 
SHOULD  DECREASE 
2:1 

NO  DATA  BASE 
NO  DIRECT  BENEFIT  FACTORS 
TAILORED /UN-CERT  AM 
UNCERTAIN  INVESTMENT 
FEARED  MUCH  GREATER 


-  CONTRACT  REWARDS  - 

-  SPEC  PHILOSOPHY  - 

-  ENOR. APPROACH  - 

'COMMUNITY"  ATTITUDES 


NO  OBVIOUS  BENEFITS 

PERFORMANCE 

INNOVATIVE 


TEST  ONLY  -  TEST  LAB  FUNCTION  -  RDTDE 

SMALL,  INDEPENDENT  -  TEST  LAS  SIZE  -  LARGE 

STATISTICS.  COSTING,  SCM.  -  PERSONNEL  BACKGROUNDS  -  ENOR.  PHYSICS.  TEST 
PMO  TEST  EG.  MFG'RS 


"MANAGEMENT"  ASSESSMENT 


•  CONTINUE  CERT  PROGRAM  DEVELOPMENT 

•  SPECIFY  USE  OF  CERT  AND  NEW-LOOK  IN 
RFP*S  AND  CONTRACTS 

•  START  REQUIRING  SOME  RELIABILITY 
GROWTH  W/CERT  IN  VALIDATION 

•  ADOPT  A  TOP  DOCUMENT  ON  "NEW-LOOK" 
MANAGEMENT 

•  WOULD  LIKE  TO  SEE  "NEW-LOOK" 
DEVELOPMENT  PLAN 


SUMMARY  OF  PRESENTATION 
by 

Colonel  Thomas  Musson 

Office  of  Under-Secretary  of  Defense/Research  and  Engineering 
Department  of  Defense 
The  Pentagon/Washington,  DC 


Colonel  Musson  addressed  the  Workshop  concerning  the  Department  of 
Defense  perspective  on  CERT  as  reflected  In  DOD  Directive  5000.40,  and  more 
recently,  In  the  recommendations  of  Mr.  Frank  Carluccl.  He  emphasized  the 
need  to  understand  the  life  profile  of  a  system,  to  define  the  environmen¬ 
tal  conditions  and  Influence  the  design  accordingly  to  do  CERT,  and  to  use 
laboratory  test  data  for  RAM  estimates  only  to  the  degree  that  the  use 
environment  Is  represented  In  the  test.  Reliability  test  objectives  are 
to:  (1)  identify  deficiencies;  (2)  provide  realistic  RAM  measures  for 
users  of  the  system;  and  (3)  determine  contractor  compliance  with  RAM 
requirements.  The  tests  defined  In  MIL-STD-785B  were  discussed  relative  to 
these  objectives. 

Colonel  Musson  concluded  his  remarks  with  a  discussion  of  his  personal 
opinion  concerning  the  factors  which  influence  system  reliability  (design, 
parts,  workmanship,  operational  concept  and  environment,  maintenance  con¬ 
cept  and  environment)  and  the  relevance  of  testing  In  the  different  phases 
of  the  system  life-cycle  to  these  factors. 

Copies  of  the  presentation  vlewgraphs  are  Included  In  the  following 
pages. 
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DoD  DIR.  5000.40  RDM  POLICY 


•  ESTABLISH  APPROPRIATE  RErM  REQUIREMENTS  FOR  EACH  ITEM. 
BASED  ON  A  DEFINED  ITEM  LIFE  PROFILE  THAT  INCLUDES 
ENVIRONMENTAL  STRESSES  AND  THE  SKILL  LEVELS  OF  OPERATOR 
AND  MAINTENANCE  PERSONNEL 

•  TEST  CONDITIONS  AND  PROCEDURES  SHALL  BE  OPERATIONALLY 
REALISTIC.  AND  THEY  SHALL  BE  DEFINED  EARLY  ENOUGH 

TO  INFLUENCE  ITEM  DESIGN 

•  PERFORMANCE,  RELIABILITY.  AND  ENVIRONMENTAL  STRESS  TESTING 
SHALL  BE  COMBINED.  AND  TYPES  OF  ENVIRONMENTAL  STRESS  WILL 
BE  COMBINED  INSOFAR  AS  PRACTICAL 

•  IN  EVERY  CASE.  MEASURED  TEST  RESULTS  SHALL  BE  CONSIDERED 
VALID  RBM  INFORMATION  ONLY  TO  THE  DEGREE  THAT  THE  TEST 
CONDITIONS  AND  PROCEDURES  SIMULATE  THE  OPERATIONAL  LIFE  OF 
A  PRODUCTION  ITEM 


OBJECTIVES 
OF  RELIABILITY  TESTING 

•  DISCLOSE  DEFICIENCIES  IN  ITEM 
DESIGN,  MATERIEL  AND  WORKMANSHIP 

•  PROVIDE  MEASURED  RELIABILITY 
DATA  AS  INPUT  FOR  ESTIMATES 

OF  OPERATIONAL  READINESS,  MISSION 
SUCCESS,  MAINTENANCE  MANPOWER 
COST  AND  LOGISTICS  SUPPORT  COST 

•  DETERMINE  COMPLIANCE  WITH 
QUANTITATIVE  RELIABILITY 
REQUIREMENTS 


TEST  REALISM  MIL-STD-785B 

•  A  TEST  THAT  ONLY  DISCLOSES  A  SMALL 
FRACTION  OF  THE  OPERATIONAL 
FAILURES  IT  IS  SUPPOSED  TO  DISCLOSE 
IS  A  WASTE  OF  TIME  AND  RESOURCES 

•  A  TEST  THAT  INDUCES  FAILURES 
WHICH  WILL  NOT  OCCUR  IN  SERVICE. 
FORCES  UNNECESSARY  EXPENDITURES 
OF  TIME  AND  RESOURCES  TO  CORRECT 
THOSE  FAILURES 

•  THE  DEGREE  TO  WHICH  ANY 
RELIABILITY  TEST  MUST  SIMULATE 
FIELD  SERVICE  DEPENDS  ON  THE 
PURPOSE  OF  THE  TEST 


MIL-STD-785B 
TEST  PROGRAM 

RELIABILITY  ENGINEERING  TESTS 

TASK  301  ENVIRONMENTAL  STRESS 
SCREENING  (ESS) 

TASK  302  RELIABILITY  DEVELOPMENT/ 
GROWTH  TEST  (RDGT) 

RELIABILITY  ACCOUNTING  TESTS 

TASK  303  RELIABILITY  QUALIFICATION 
TEST  (RQT) 

TASK  304  PRODUCTION  RELIABILITY 
ACCEPTANCE  TEST  (PRAT) 
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MIL-STD-785B 
TEST  PROGRAM 


TASK  TEST  APPROACH 
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RETHINKING  CERT:  A  REALISTIC  PERSPECTIVE 


Henry  Caruso 
Senior  Engineer 

Westlnghouse  Electric  Corporation 
Baltimore,  Maryland 


A  Technological  Orphan 

Combined  environment  testing  Is  not  new;  tests  using  combinations  of  a 
variety  of  forcing  functions  have  been  used  for  decades.  In  1975,  the 
Air  Force  Flight  Dynamics  Laboratory  at  Wright-Patterson  AFB  Introduced  the 
concept  of  accurately  combining  measured  flight  environmental  conditions 
for  mission  profile  testing  In  the  laboratory  to  the  technical  community  at 
large.  (Ref.  1).  This  method  was  called  CERT,  for  Combined  Environment 
Reliability  Testing.  Since  that  time,  numerous  studies  have  been  made  and 
reports  distributed.  CERT  methodology  for  mission  profile  testing  was 
Included  In  MIL-STD-781C  (Ref.  2)  nearly  four  years  ago  and  Is  recognized 
In  current  DoO  policy  (Ref.  3).  The  draft  versions  of  MIL-STD-810D  and 
MIL-STD-781D  both  Include  combined  environment  testing. 

Yet,  In  spite  of  constant  and  widespread  exposure,  combined  with  offi¬ 
cial  DoD  recognition,  CERT  has  still  not  achieved  anything  resembling  con¬ 
sistent  or  enthusiastic  acceptance.  There  Is  no  agreed  upon  approach  to 
the  systematic  application  of  CERT  methodology.  Even  the  effectiveness  and 
benefits  of  CERT  remain  unresolved  within  the  same  community  of  pro¬ 
fessionals  that  promoted  the  development  of  CERT  more  than  six  years  ago. 

In  the  Interest  of  establishing  a  direction  for  dealing  with  these 
circumstances,  this  paper: 

1.  Analyzes  the  underlying  causes  of  the  ambivalent  attitude 
shown  CERT  to  date. 


of  CERT. 


2.  Objectively  evaluates  the  potential  benefits  and  limitations 


3.  Recommends  several  logical  approaches  to  CERT  application. 


CERT  is  a  Four-Letter  Word 


Certain  four-letter  words  have  acquired  a  special  notoriety  because  of 
their  dual  identities.  They  not  only  serve  as  labels  for  objects  or 
actions,  but  also  evoke  strong  emotional  reactions  based  on  personal  sen¬ 
sitivities  and  background. 

In  referring  to  CERT  as  a  four-letter  word,  I  do  not  mean  to  Imply  a 
value  judgment  of  any  kind  regarding  CERT's  acceptability  or  value. 

However,  CERT  too,  is  characterized  (unfortunately  to  Its  disadvantage)  by 
a  similar  dual  Identity.  On  the  one  hand,  CERT  serves  simply  as  a  label 
for  a  technological  tool.  On  the  other  hand,  CERT  seems  to  have  the  abi¬ 
lity  to  trigger  strong  subjective  responses  at  Its  mention.  It  is  quite 
possible  that  these  reactions  are  an  attempt  to  fill  the  judgmental  vacuum 
left  by  the  professional  community.  But  regardless  of  the  reason  for  these 
emotional  reactions,  they  only  serve  to  further  cloud  the  issue  of  CERT 
val idity. 

In  the  following  discussion,  I  would  like  to  distinguish  between  the 
objective  and  subjective  Identities  of  CERT  and  treat  each  separately. 
First,  let's  consider  CERT  as  a  label. 


CERT  Defined 


I  would  like  to  recommend  adoption  of  the  following  definition  of  CERT: 
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"Any  laboratory  test  for  hardware  reliability  Improvement 
or  characterization*  In  which  environmental  forcing  functions 
are  applied  simultaneously." 

This  straightforward  definition  is  free  from  any  suggestion  of  program 
phase,  contractual  motivation,  specific  methodology,  or  facility  sophisti¬ 
cation.  It  allows  CERT  to  be  viewed  objectively  as  just  another  tool 
available  to  the  test  designer  for  hardware  development.  As  such,  It  pro¬ 
vides  a  logical  starting  point  from  which  to  begin  a  CERT  evaluation. 


CERT  Myths 

Before  discussing  what  CERT  Is,  it  Is  Important  to  first  clarify  what 
it  is  not.  Certain  myths  concerning  CERT  often  seem  to  be  treated  as  facts 
In  CERT  debates.  These  myths  have  not  been  deliberately  perpetrated  by  any 
specific  Individual  or  organization.  Rather,  they  seem  to  result  from 
Incomplete  communications.  Nevertheless,  the  result  Is  that  many  arguments 
advanced  both  for  and  against  CERT  are  without  credible  foundation. 

Several  of  these  myths  are  briefly  described  below. 

Myth  No.  1 :  CERT  Is  limited  to  certain  forcing  functions. 

With  the  definition  recommended  above  as  a  basis,  there  Is 
obviously  no  limitation  placed  upon  forcing  function  types  or  nuntoers  by 
the  CERT  concept.  Such  terms  as  "Full-CERT"  and  "CERT-Without  (a  given 
forcing  function)"  only  tend  to  limit  CERT's  potential  uses  by  Implying 
the  existence  of  an  "official"  baseline  combination  of  forcing  functions. 


Note:  *The  provision  of  measured  reliability  data  as  Input  for  estimates  of 
operational  readiness,  mission  success,  maintenance  manpower  cost  and 
logistics  support  cost  (Ref.  15). 
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Myth  No.  2:  CERT  *  Mission  Profile  Testing. 


While  mission  profile  testing  is  undeniably  a  major  subset 
of  CERT,  it  is  wrong  to  assume  that  no  other  forms  of  CERT  exist.  For 
example,  CERT  may  be  used  quite  effectively  for  stress  screening  by 
tailoring  the  forcing  functions  to  specific  defect  modes  or  hardware 
characteristics  (Ref.  4),  regardless  of  mission  forcing  function  profiles. 

To  evaluate  Its  full  utility,  then,  CERT  must  be  considered  in  all  of  its 
possible  manifestations  and  not  artificially  limited  to  one  specific 
methodology. 

Myth  No.  3:  CERT  makes  troubleshooting  and  failure  difficult. 

Some  have  taken  the  position  that  troubleshooting  and 
failure  analysis  are  made  more  difficult  when  more  than  one  environmental 
forcing  function  Is  applied  at  a  time.  However,  actual  CERT  users' 
experience  (Ref.  5)  has  shown  no  such  difficulty,  with  troubleshooting  being 
accomplished  by  normal  means  while  removing  or  using  one  forcing  function 
at  a  time.  Established  failure  analysis  techniques  still  provide  infor¬ 
mation  useful  in  determining  appropriate  corrective  action.  Once  a  correc¬ 
tive  action  has  been  Identified,  the  specific  environmental  forcing 
function  responsible  for  the  failure  becomes  academic. 

Myth  No.  4:  CERT  facilities  are  too  expensive. 

This  is  a  common  assertion,  apparently  based  on  the  mistaken 
assumption  that  certain  complex  experimental  facilities  are  representative 
of  the  way  all  CERT  facilities  must  be.  While  it  is  true  that  it  will  cost 
more  to  combine  and  tailor  some  forcing  functions  than  others,  this  cost  is 
only  too  expensive  when  the  test  benefits  are  not  commensurate  with  the 
facility  expense.  In  fact,  expensive  testing  (as  related  to  test  benefits) 
is  really  the  consequence  of  administrative  and  regulatory  policies  which 
rigidly  and  unnecessari ly  constrain  Its  use.  CERT  doesn't  have  to  i. 
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prohibitively  expensive,  but  It  Is  all  too  often  artificially  driven  to  be 
so  by  those  assuming  the  previously  stated  nyths  to  be  fact. 


Myth  No.  5:  CERT  facilities  are  few  and  far  between. 

Combined  environment  test  facilities  have  been  In  use  for 
decades.  In  the  past  several  years,  many  companies  and  government  organi¬ 
zations  have  purchased  or  developed  CERT  facilities  with  extensive  mission 
profile  test  capabilities.  Some  of  these  have  been  purchased  directly  from 
commercial  facility  vendors  while  others  have  been  developed  from  scratch 
or  from  modified  existing  facilities  by  the  organization's  own  personnel . 
Forcing  function  types  and  control  options  span  a  broad  range.  The  main 
point  Is  that  CERT  facilities  are  no  longer  the  laboratory  curiosity  they 
once  were.  They  are  used  routinely  by  a  growing  number  of  organizations  for 
a  wide  variety  of  hardware  development  and  acquisition  purposes  (Refs. 
6,7,8,9).  This  expanding  facilities  base  can  serve  as  a  source  of  valuable 
CERT  experience. 


A  Multi-Variable  Experiment 

CERT’s  existence  to  date  as  a  mission  profile  test  methodology  can 
best  be  described  as  an  uncontrolled  experiment.  The  differences  between 
mission  profile  CERT  and  traditional  environmental  tests  go  far  beyond  the 
simple  combination  of  standardized  single-environment  forcing  functions. 
Mission  profile  CERT  In  reality  has  been  an  experiment  In  which  several 
variables  have  been  changed  at  once,  thereby  complicating  analysis  of  the 
experimental  results.  To  date,  this  experiment  has  Introduced  the 
following  variables: 

1.  Forcing  Function  Combination:  Forcing  functions  applied  In  com¬ 
bination  Instead  of  sequentially. 


2.  Tailored  Forcing  Functions:  Forcing  function  profiles  tailored  to 
those  expected  or  measured  in  actual  service,  commonly  referred  to  as 
mission  profiles.  This  contrasts  with  traditional  sequential  testing  which 
Is  normally  based  upon  untailored,  standard  parameters  and  methods. 

3.  Forcing  Function  Repetition:  In  mission  profile  CERT,  forcing 
functions  are  often  repeated  for  a  given  number  of  equivalent  missions 
(cycles).  Further,  one  hardware  item  is  exposed  to  all  forcing  function 
types..  Traditional  sequential  tests,  on  the  other  hand,  are  generally 
applied  only  one  time,  thereby  discouraging  the  full  evaluation  of  cumula¬ 
tive  stress  effects.  In  addition,  traditional  test  programs  often  use 
several  hardware  items  tested  in  parallel  to  different  forcing  function 
types.  The  result  is  that  no  single  hardware  item  is  exposed  to  all 
forcing  function  types. 

4.  Scoring:  Failure  definition  (or  acknowledgement)  ground  rules  for 
mission  profile  CERT  and  traditional  tests  have  not  always  been  the 

same  (Ref.  10). 

The  result  is  that  the  improved  problem  disclosure  attributed  to  mission 
profile  CERT  could  be  due  to  any  of  these  variables,  three  of  which  do  not 
involve  combining  environments. 


CERT  Utility 


To  this  point,  the  mechanisms  responsible  for  the  potential  benefits 
of  CERT  do  not  appear  to  be  well  characterized.  While  this  circumstance  is 
certainly  not  an  indictment  of  the  concept,  it  does  present  a  serious 
obstacle  to  its  widespread  acceptance. 

Fortunately,  however,  there  is  a  much  more  important,  yet  commonly 
overlooked,  standard  against  which  to  evaluate  CERT:  efficiency.  More 


specifically,  any  problems  that  can  be  disclosed  by  any  one  of  the  applied 
forcing  functions  will  be  disclosed  by  the  same  test  at,  In  essence,  the 
same  time.  This  Is  not  synergism.  Rather,  this  Is  parallel  tests  with 
different  forcing  functions  applied  to  the  same  hardware  Item.  The  result 
Is  earlier  disclosure  of  problems  and,  therefore,  the  best  opportunity  for 
accelerated  Improvement  of  hardware  quality  (In  effect,  accelerated 
reliability  growth).  Government  and  Industry  experience  (Refs.  11,14)  has 
shown  that  significant  time  and  cost  savings  caO  be  realized  (Figure  1). 


TESTS 

SAVINGS 

Dollars 

Days 

Deleted  7  separate  environmental 

qualification  tests 

$195,400 

129 

Deleted  reliability  demonstration 

test  (5,000  hrs  0  $1 70/hr) 

850,000 

125 

Replaced  with  a  single  CERT  growth  test 

230,000 

50 

Total  Savings 

$815,400 

204 

Figure  1.  Savings  Available  Using  a  CERT  Growth  Test  (Ref.  14) 


In  this  context,  then,  CERT  may  be  thought  of  as  schedule  compression; 
as  If  several,  normally  sequential,  single  forcing  function  tests  were 
applied  to  the  same  hardware  In  parallel.  The  benefits  of  this  schedule 
compression  can  be  greatest  for  those  hardware  development  and  acquisition 
phases  (engineering  development  and  stress  screening)  which  are  often 
emphasized  least  In  program  planning  in  the  drive  to  ensure  that  compliance 
(i.e.,  qual ification,  acceptance)  test  requirements  are  satisfied.  The 
following  section  will  discuss  these  benefits  at  greater  length. 

The  Troublesome  "R" 


The  main  source  of  controversy  over  CERT  seems  to  be  the  "R",  Reliabi¬ 
lity.  CERT  is  not  a  traditional  reliability  approach  and  therefore  lacks 
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the  statistical  pedigree  needed  for  acceptance  within  some  reliability 
circles.  As  a  result,  reactions  to  CERT  within  the  reliability  community 
have  been  observed  to  range  from  casual  Indifference  to  hostile  opposition. 
However,  CERT  Is  also  valuable  for  reliability  Improvement  through  defi¬ 
ciency  disclosure  (Ref.  15)  and  therefore  must  not  be  Ignored.  If  a  sin¬ 
cere  effort  Is  made  to  carefully  and  intelligently  apply  CERT  to  a 
representative  hardware  population,  the  appropriate  statistical  charac¬ 
terization  will  logically  follow. 

Several  key  areas  In  which  CERT  can  be  of  value  to  reliability  testing 
merit  consideration  and  are  as  follows: 

1.  Growth  Testing:  The  rate  at  which  hardware  improvement  occurs 
will  be  directly  influenced  by  the  rate  at  which  problems  are  disclosed. 
Combined  forcing  function  tests  can  disclose  these  problems  faster  (see 
Figure  1)  than  traditional  sequential  tests  and,  therefore,  offer  great 
promise  for  accelerating  reliability  growth  (Ref.  11,14). 

2.  Reliability  Characterization:  The  potential  for  test  realism 
using  mission  profile  CERT  can  be  used  to  improve  the  accuracy  and  thus  the 
usefulness  of  hardware  characterization.  This  improved  accuracy  should  be 
of  particular  value  in  spares  and  logistics  planning.  (While  not  currently 
a  common  practice,  characterization  could  be  made  more  efficient  by  a  form 
of  time  compression  In  which  only  the  forcing  function  levels  most  likely 
to  disclose  problems  are  applied;  benign  levels  are  not  used.  Corres¬ 
pondence  between  laboratory  and  field  exposure  to  these  forcing  functions 
could  be  retained  with  appropriate  correction  constants.) 

3.  Stress  Screening:  When  hundreds  or  thousands  of  hardware  Items  are 
involved,  any  reduction  in  the  time  required  to  screen  each  Item  will  be 
multiplied  many  times  over  with  potentially  enormous  dollar  savings  (Figure  2). 
Stress  screening  CERT,  tailored  to  defect  type  and  hardware  anatomy  (Ref.  4) 

(as  opposed  to  mission-use  environmental  profiles)  and  applied  by  Inexpen¬ 
sive  alternative  forcing  function  generators,  such  as  pneumatic  vibrators 
(Ref.  12)  and  portable  temperature  enclosures,  can  provide  this  time  reduc¬ 
tion  with  minimal  facility  cost  penalties. 
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Vibration  Testing  In  One  Axis: 

Test  setup  45  minutes 

Vibration  10  minutes 

Performance  Check  45  minutes 

Total  100  minutes/unit 


If  performed  as  a  separate  test,  vibration  would  add  at  least  100 
minutes/per  unit  to  the  overall  time  required  for  screening.  If  a 
representative  production  run  of  1000  units  Is  assumed,  the  savings 
associated  with  combining  environments  are: 

Screening  CERT  Time  Savings  =  (100  mln/unlt)  (1000  units) 
(Assuming  2  shlfts/day)  (60  mln/hour)  (16  hrs/day) 

=  104+  Days  (without  parallel  test) 

Screening  CERT  Cost  Savings  =  (100  mln/unlt)  (1000  units)  (S/hr) 
($25-$35/hr  range)  (60  mln/hr) 

=  $41,666  to  $58,333 


Assumption  1  -  Screening  performed  on  small  to  moderate  size 
electronics  boxes. 

Assumption  2  -  At  least  one  other  environmental  forcing  function 
combined  with  vibration. 

Assumption  3  -  Environmental  forcing  function  combined  with  vibration 
takes  at  least  as  long  to  apply  as  vibration.  In  the 
case  of  thermal  cycling,  this  Is  a  safe  assumption. 

Assumption  4  -  Only  one  axis  of  vibration  was  assumed.  If  2  or  3 
orthogonal  axes  are  used,  screening  time  and  cost 
differentials  are  multiplied  by  these  factors. 


Figure  2.  Example  of  Potential  Time  and  Cost-Savings 
Associated  with  Stress  Screening  CERT. 
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Building  on  Strength 


Key  recommendations  made  early  In  the  Wright-Patterson  AFB  CERT  study 
program  (Ref.  13)  are  still  valid.  Unfortunately,  fulfillment  of  these 
goals  in  ways  supported  by  widespread  concurrence  has  yet  to  happen. 

These  recommendations  are: 

1.  Develop  a  standard  method  of  deriving  environmental  profiles 

2.  Balance  the  engineering  benefits  of  more  complete  environmen¬ 
tal  simulation  against  Increased  facility  complexity  and  cost 

3.  Establish  a  data  base  for  the  complete  evaluation  of  CERT 

4.  Develop  a  time-accelerated  CERT  methodology 

CERT  experiments  In  Industry  and  government  to  date  have  been  Incon¬ 
sistently  supported  and  Incompletely  defined,  but  have  nevertheless  pro¬ 
duced  valuable  data.  It  Is  now  time  for  the  professional  community  to  try 
to  determine  what  these  data  are  telling  us.  However,  as  with  any  scien¬ 
tific  experiment,  we  must  be  prepared  to  accept  any  legitimate  results, 
even  though  they  may  not  agree  with  our  original  expectations.  Otherwise, 
there  is  a  strong  risk  of  being  blind  to  useful  information. 

Although  not  rigorously  defined  or  quantified,  there  are  significant 
potential  benefits  to  be  derived  from  the  systematic  application  of  CERT  to 
hardware  development  and  acquisition.  The  challenge  we  now  face  Is  to 
agree  upon  clearly  defined  processes  for  applying  CERT  to  satisfy  differing 
reliability  test  needs.  These  processes  should  allow  already  existing 
expertise  and  facilities  to  be  put  to  their  best  use.  The  talent  to 
achieve  this  goal  Is  available.  However,  building  on  this  strength  can 
only  take  place  In  an  environment  characterized  by  commonality  of  purpose, 
technical  honesty,  and  willing  cooperation.  Providing  this  combination  of 
environments  will  be  the  biggest  challenge. 
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A  Closing  Observation 

Those  experienced  In  environmental  testing  recognize  that  a  require¬ 
ment  such  as  "perform  a  vibration  test"  provides  no  usable  guidance.  A 
vibration  test  could  use  random,  sinusoidal,  acoustic,  continuous  or 
pulsed  excitation,  and  could  be  based  on  Input-control  or  response- 
definition,  orthogonal  axis  or  diagonal  vector  energy  Input.  In  short, 
there  Is  no  functional  meaning  for  the  term  vibration  test  In  the  absence 
of  appropriate  qualifiers. 

CERT  as  a  concept  suffers  from  the  same  Incompleteness  of  definition. 

CERT  can  legitimately  exist  as  a  large  number  of  different  methodologies 
and  procedures,  to  the  potential  confusion  of  all  Involved.  While  the 
technical  characterization  of  CERT  types,  benefits  and  liabilities  remains 
as  an  Important  engineering  task,  the  consistency  and  clarity  of  our  com¬ 
munications  on  the  subject  might  be  aided  by  the  simple  expedient  of 
adopting  the  right  clarifiers.  Suggested  examples  Include: 

1 .  Growth  CERT 

2.  Qualification  CERT 

3 .  Devel  opment  CERT 

4.  Stress  Screening  CERT 

5.  Mission  Profile  CERT 

While  not  a  solution  to  the  problem  at  hand,  such  an  approach  could 
remove  a  number  of  communication  obstacles  which  will  otherwise  make  the 
process  of  arriving  at  an  accepted  solution  much  more  arduous  than  It  need  be. 
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ABSTRACT 


The  primary  purpose  of  this  paper  Is  to  delineate  and  emphasize  the 
environmental  engineering  functions  that  are  necessary  during  a  hardware 
development  program  to  achieve  design  Integrity  and  operational  reliability 
and  readiness.  Involvement  of  environmental  engineering  functions  Is  shown 
to  be  necessary  earlier  than  at  the  time  of  environmental  testing  during  a 
development  program  If  optimum  life-cycle  costs  are  to  be  achieved. 
Suggestions  are  made  as  to  what  these  necessary  functions  are  and  at  which 
program  phase  they  should  be  Invoked.  A  trend  of  reliability  Improvement 
with  environmental  testing  is  shown.  Seven  fundamental  types  of  tests, 
distinguished  as  to  purpose  and  type  of  test,  are  defined,  and  their 
characteristics  are  summarized.  Recommendations  are  made  for  incorporating 
descriptions  of  the  environmental  engineering  functions  In  procurement 
documents  and  for  policing  application  of  the  functions  by  means  of 
contractor-prepared  and  customer-approved  reports  at  various  program  pha¬ 
ses.  The  paper  draws  on  recommendations  which  originated  within  working 
groups  of  the  IES,  particularly  within  IES  publications  by  the  author.  One 
of  those  publications  In  the  Journal  of  Environmental  Sciences,  In 
September  1979  (Reference  1),  had  a  title  similar  to  this  paper’s  title, 
and  one  on  proposed  IES  Recommended  Test  Practices  was  published  In  the  IES 
Technical  Symposium  Proceedings  In  May  1981  (Reference  2). 
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INTRODUCTION 


Results  of  studies  of  reliability  of  avionics  by  the  Department  of 
Defense  In  the  early  1970's  placed  a  great  emphasis  on  the  need  for  more 
realistic  environmental  design  and  testing  to  Improve  the  survival  of  equip¬ 
ment  In  operating  environments  (See  Reference  3  and  4,  for  example). 
Conclusions  from  these  and  subsequent  studies  can  be  paraphrased  as  follows: 
"Failure  to  design  and  test  adequately  for  realistic  operational  environments 
accounts  more  than  any  single  factor  for  the  failure  to  achieve  desired 
reliability  in  mil Itary- fielded  equipment".  While  a  number  of  hardware 
testing  procurement  documents  have  been  revised  to  reflect  the  concern  with 
realistic  test  environments,  a  similar  emphasis  has  not  been  added  to  program 
"front  end"  engineering  (which  sets  the  requirements  for  design  and  calls 
out  testing  methods).  This  may  be  attributed  to  two  primary  factors  which  are 
common  to  both  the  organization  that  designs  and  produces  a  system  and  to  the 
procurement  agency:  (1)  failure  to  recognize  that  the  environmental  engineer 
normally  accesses  a  program  only  through  a  document  such  as  MIL-STD-810,  which 
Is  not  Invoked  until  late  In  full-scale  development,  and  (2)  belief  that 
published  documents  and/or  procurement  documents.  In  general,  adequately  spe¬ 
cify  environments  for  design  and  test.  The  latter  conflicts  with  the  current 
requirement  In  Department  of  Defense  Directive  5000.40  for  tailoring  pro¬ 
curement  documents  to  the  particular  piece  of  equipment.  Although  some 
tailoring  can  be  done  by  environmental  spe~1al1sts  within  the  procurement 
agency.  It  Is  probable  that  only  the  designing  and  producing  organization  has 
enough  information  (such  as  transfer  functions  available  for  larger  systems) 
to  permit  tailoring  of  general  environments  to  a  particular  box  at  a  given 
location  for  timely  utilization  in  the  program  to  affect  design  and  test. 
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DISCUSSION 


Phase  and  Cost  Elements  of  a  Program 

For  procurement  of  equipment  that  Is  to  be  added  to  an  existing  vehicle, 
it  may  be  possible  for  the  procuring  activity  to  specify  levels  of  environ¬ 
ments  that  are  based  on  measurements  from  the  actual  vehicle.  However,  in  the 
case  of  the  entire  vehicle  or  major  component,  such  as  a  wing-mounted  store, 
the  designing  and  producing  organization  is  faced  with  this  task.  Procurement 
documents  such  as  MIL-STD-810C  only  invoke  environmental  engineering  methods 
during  full-scale  development.  At  the  time  that  such  documents  normally  are 
applied.  It  is  too  late  in  the  program  to  be  effective  in  changing  major 
program  decisions,  such  as  choices  on  location  of  equipment  and  basic  layout 
of  equipment  bays.  Such  decisions  contribute  to  life-cycle  costs  and  severity 
of  environment.  Figure  1  Illustrates  this  point.  It  was  constructed  from 
Information  that  was  obtained  from  a  JLC  Design-To-Cost  Guide  (Reference  5). 
The  figure  shows  that  85  percent  of  the  decisions  that  affect  life-cycle  costs 
are  made  by  the  end  of  system  definition.  An  additional  10  percent  of  the 
decisions  are  made  during  full  scale  development.  Therefore,  it  would  be 
highly  desirable  to  affect  environmental  design  (to  the  extent  that  it  affects 
life-cycle  costs)  during  the  concept  studies.  In  particular,  the  environmen¬ 
tal  trade  studies  of  equipment  locations  and  of  life-cycle  environments  should 
oe  accomplished  prior  to  DSARC  I  for  maximum  effectiveness.  The  curve  in 
Figure  1  does  not  Imply  that  the  actual  work  is  done  prior  to  the  points 
shown;  It  indicates  only  that  the  decisions  which  affect  what  is  done  are  made 
by  times  that  are  shown.  As  an  example,  the  decision  to  conduct  reliability 
development  tests  during  full-scale  development  should  be  made  during  concept 
studies. 

Figure  2  which  was  extracted  from  Reference  6,  provides  a  point  of 
reference,  illustrating  the  times  and  elements  that  are  involved  in  the  deve¬ 
lopment  of  a  system.  On  the  bottom  of  Figure  2,  a  bar  has  been  added  to 
Indicate  a  typical  number  of  years  required  to  achieve  certain  phases.  The 
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bar  has  been  added  to  indicate  a  typical  number  of  years  required  to  achieve 
certain  phases.  The  bar  also  indicates  typical  contractor  activities  which 
relate  to  environmental  and  reliability  engineering  and  testing  during  those 
phases.  This  figure  Illustrates  some  of  the  overlap  between  phases,  and  the 
difficulty  In  defining  the  exact  phase  when  something  should  be  accomplished 
on  a  given  program.  For  any  particular  program,  much  greater  overlaps  may 
occur.  For  example,  full-scale  development  may  continue  until  IOC.  As  Indi¬ 
cated  on  the  bottom  of  the  figure,  reliability  demonstration  tests,  if 
required,  may  continue  until  long  after  the  system's  reliability  has  been 
demonstrated,  and  may  even  extend  well  past  full-scale  production  go-ahead, 
during  customer  TAE.  The  activities  which  are  listed  on  the  bottom  of  the 
figure  Illustrate  two  additional  points:  (1)  reliability  goals  typically  are 
established  in  the  validation  phase  at  the  time  of  full-scale  development  pro¬ 
posal  (just  under  the  point  labeled  six  years),  during  which  time  the  basic 
motivation  of  the  contractor  Is  to  pass  the  validation  tests;  (2)  passing  the 
validation  tests  requires  that  the  contractor  use  the  limited,  available  6.3 
money  to  conduct  only  those  tests  which  will  assure  adequate  demonstration  of 
concept  validation.  It  would  appear  then,  that  If  we  are  to  Impact  decisions 
that  affect  life-cycle  costs,  the  procuring  activity  may  need  to  fund  some 
environmental  testing  and  engineering  during  the  first  six  years  (validation 
phase)  even  though  It  means  some  duplication  if  two  or  more  contractors  are 
Involved.  Such  funding  may  be  very  effective  in  lowering  life-cycle  costs. 

Environmental  Engineering  Functions 

It  is  recognized  that  recently  revised  procurement  documents  such  as 
MIL-ST0-785B  and  MIL-ST0-781C  (References  7  and  8)  enumerate  some  of  the 
requirements  for  environmental  engineering  and  testing.  It  Is  also 
understood  that  MIL-STD-810D  (Reference  9),  which  is  currently  In  the  revi¬ 
sion  stage,  will  require  that  some  primary  engineering  efforts  precede  spe¬ 
cified  tests.  However,  none  of  these  documents  Identifies  an  "environmental 
engineering  discipline",  so  they  do  not  clarify  the  purposes  and  tasks  that 
are  to  be  accomplished  by  this  discipline;  rather  the  function  Is  treated  as  a 
subset  of  testing  or  of  reliability  and  maintainability  engineering.  Although 
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this  may  prove  satisfactory,  an  attempt  was  made  in  Reference  1  to  segre¬ 
gate  the  environmental  engineering  discipline  for  the  purpose  of  defining 
objectives  and  tasks  that  are  necessary  to  accomplish  the  objectives  that  are 
embodied  in  the  "new-look". 

If  we  were  to  treat  environmental  engineering  as  a  function  that  is  similar 
to  the  reliability  function  that  is  established  in  MIL-STD-785,  then  the  objec¬ 
tives  for  the  environmental  engineering  function  probably  would  be  as  given  in 
Table  I.  Since  environments  that  are  generated  by  the  product,  particulary  in 
the  case  of  an  entire  vehicle,  affect  the  surrounding  environments,  the  third 
technical  objective  that  is  listed  in  Table  I  would  be  a  necessary  part  of  the 
environmental  engineering  discipline.  This  activity  normally  is  relegated  to 
the  manufacturi ng  department,  but  it  is  becoming  much  more  relevant  to  the 
environmental  discipline  with  the  advent  of  regulations  that  have  been 
established  by  the  Environmental  Protection  Agency  and  the  Occupational  Safety 
and  Health  Administration;  recently,  the  Surgeons  General  have  issued  similar 
regulations  for  the  military  services. 

Table  II  lists  the  functions  of  the  environmental  discipline  that  arise 
from  the  objectives  that  were  just  established.  The  first  three  items 
(functions  in  Table  I)  are  roughly  in  chronological  order.  The  fourth  might  be 
in  sequence  for  major  procurements,  but  it  may  occur  first  for  additions  to 
existing  platforms.  In  any  event.  Information  that  is  obtained  during  any  one 
of  thi  functions  should  be  examined  for  possible  feedback  or  input  to  the  other 
Tun. lions.  The  first  four  of  the  functions  that  are  listed  in  Table  II  might 
Orove  effective  if  they  were  presented  in  current  testing  standards  as  necessary 
prerequisi tes  to  defining  life-cycle  environments  for  test  purposes. 

The  five  primary  environmental  engineering  program  functions  can  be  sub¬ 
divided  into  engineering  work  tasks.  These  are  listed  in  Table  III  and  segre¬ 
gated  roughly  by  major  program  phase:  research,  development,  testing  and 
ova^ation.  The  fi 'st  seven  of  these  tasks  are  necessary  prerequisites  to 
testing,  and  they  occur  in  chronological  order.  The  next  seven  tasks,  which 
also  appear  in  chronological  order,  identify  seven  basically  different  types 
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of  tests  which  are  described  in  Reference  2.  A  given  procurement  may  not 
require  all  of  those  tests.  The  last  four  tasks  should  be  accomplished  for  any 
given  program.  They  relate  primarily  to  evaluation  of  effectiveness  of  the 
environmental  engineering  and  test  functions. 

Environmental  Testing  Functions 


Combined  Environment  Testing  is  a  necessary  component  of  the  “new-look". 
There  has  been,  and  will  continue  to  be,  an  argument  as  to  whether  the  acronym 
CERT,  for  Combined  Environment  Reliability  Testing,  or  CET,  for  Combined 
Environment  Testing,  should  be  used  to  describe  the  tests.  Strictly  speaking, 
only  those  tests  which  have  a  reliability  measure  output  should  utilize  the  term 
CERT.  Therefore,  the  more  general  CET  has  been  utilized  within  the  IES.  In 
either  case,  it  is  difficult  for  two  people  to  communicate  the  definition  of  the 
term,  because  there  are  so  many  variables  that  are  necessary  for  definition  of 
the  term.  Four  sets  of  those  variables  are  listed  in  Figure  3.  When  utilizing 
either  acronym,  it  is  a  good  idea  to  select  at  least  one  word  from  each  of  the 
columns  which  are  shown  in  Figure  3  to  assure  that  the  speaker  and  the  listener 
are  on  common  ground.  The  first  column  can  be  expanded  almost  infinitely  by 
subdividing  each  of  the  major  platforms.  In  addition,  there  is  no  general 
agreement  on  the  definition  of  the  terms  which  are  listed  in  the  third  and  fourth 
columns.  The  Institute  of  Environmental  Sciences  (IES)  is  attempting  to  elimi¬ 
nate  this  problem  by  publishing  a  series  of  recommended  practices  which  contain 
glossaries.  A  fifth  column,  which  describes  the  level  of  environment,  should  be 
added  to  this  listing.  It  would  contain  the  following  variables:  fixed  level, 
mission  or  life-cycle  profile,  some  idea  as  to  the  length  of  test  time.  If  the 
two  acronyms  are  defined  in  accordance  with  Figure  3,  then  much  unnecessary  con¬ 
fusion  can  be  alleviated. 

Figure  4  has  been  extracted  from  Reference  2  to  provide  an  example  of  the 
definitional  problem  and  of  one  of  the  solutions  preferred  by  the  IES.  It 
depicts  a  general  trend  of  reliability  Improvement  with  environmental  testing. 

The  figure  identifies  seven  basic  types  of  environmental  tests  which  are 
distinguished  by  their  purpose,  description  of  test,  and  expected  benefits  and 
costs.  The  titles  of  the  tests  in  Figure  4  are  largely  self-explanatory.  In 
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Reference  2,  the  design  proof  test  (that  Is  defined  as  2a  In  Figure  4)  was 
regarded  as  similar  enough  to  the  test  that  is  defined  as  2  to  bear  the  same 
number.  In  future  publications,  2a  probably  will  be  renumbered,  so  that  seven 
tests  will  be  listed.  A  distinction  has  been  made  here  between  those  tests  that 
are  regarded  as  RDT£E  and  those  that  are  regarded  as  production  type  tests. 
Reliability  growth  occurs  during  the  first  three  tests  but  is  not  expected  to 
occur  during  either  the  design  proof  test  or  the  reliability  demonstration  test. 
The  characteristics  of  these  tests  are  sunmarized  in  Table  IV.  Not  all  of  these 
tests  will  be,  or  should  be,  required  on  a  given  program.  Rather,  they  provide 
a  convenient  distinction  for  discussion  purposes.  Necessary  requirements  for 
all  tests  are-,  fl)  feed-back  on  failures  during  tests  and  in  the  field  to  aid 
corrections  of  design,  if  they  are  necessary;  (2)  acceptance  testing  of  all  ven¬ 
dor  parts  prior  to  incorporation  in  the  test  specimen;  and  (3)  burn-in  of  all 
specimens  to  remove  infant  mortality  and  manufacturing  defects  prior  to  testing. 

The  characterizations  of  the  various  tests  that  are  shown  in  Table  IV 
reflect  the  consensus  of  the  IES  working  group  on  environmental  and  reliability 
testing.  The  major  purpose  of  each  test  has  been  indicated  on  line  2  of  Table 
’V,  and  the  major  beneficiary  of  the  testing  operation  is  noted  on  line  3. 

As  stated  earlier,  the  purpose  of  the  IES  recommended  practice  (of 
Reference  2)  is  for  presentation  to  management  to  clarify  the  role  of  testing  in 
hardware  development  programs.  An  exampTe  of  a  related  standard  or  specifica¬ 
tion  that  has  previously  been  published  and  which  describes  a  portion  of  each  oc 
the  tests  is  given  in  line  9  of  Table  IV.  Line  13  of  the  table  reflects  an 
attempt  to  indicate  when  the  test  would  prove  most  effective  toward  reduction  of 
1 ife-cycle  costs. 

To  illustrate  the  ideas  above  and  to  illuminate  the  concept  that  peculiar 
crograms  do  not  require  all  tests,  at  all  assembly  levels,  with  all  combinations 
'r  environments,  a  fictitious  testing  urogram  summary  has  been  prepared  in  Table  V 
It  would  be  appropriate  to  a  vehicle  such  as  a  cruise  missile  or  a  ground- 
iaunced  missile,  tut  to  no  system  tnat  specifically  is  in  current  procurement, 
i  table  might  be  . p;  ;  op* '  for  ir.!_'-jsio.n  in  a  system  specification  which 
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could  be  submitted  In  the  concept  definition  phase  of  a  program,  prior  to  vali¬ 
dation.  The  parts  description  in  Table  V  is  In  accordance  with  MIL-STD-280A. 

The  design  evaluation  test  which  is  shown  on  Figure  4  has  not  been  called  out 
separately  in  Table  V,  but  it  has  been  assumed  as  a  necessary  first  step  under 
TAAF.  mere  was  considerable  argument  in  the  IES  working  group  on  the 
appropriateness  of  the  names  for  the  particular  tests,  the  particular  com¬ 
binations  of  tests,  and  the  types  of  tests  that  are  shown  in  Table  V.  A  general 
consensus  evolving  from  the  working  group  was  that  each  particular  test  on  a 
given  program  should  be  shown  to  be  cost-effective  prior  to  being  incorporated 
at  each  level  of  assembly.  Table  V  illustrates  the  difficulty  in  defining  a 
given  CERT,  if  indeed  one  does  exist;  and  it  suggests  that  while  the  concept  of 
CERT  is  valid,  its  application  is  extremely  product-peculiar. 

The  methodology  that  is  represented  in  the  "new-look"  and  which  is 
discussed  above,  represents  a  change  in  environmental  testing  and  engineering 
that  has  taken  20  years.  As  a  consequence,  It  is  not  readily  accepted  by 
program  managers  who  are  faced  with  meeting  contract  front-end  costs  and  schedu¬ 
les.  A  program  administrator  who  operates  under  these  constraints  cannot  be 
extremely  Interested  in  reductions  of  downstream  operational  support  cost.  It 
has  been  widely  assumed  that  the  additional  testing  and  the  new  methodology  that 
is  required  with  the  "new  look"  would  require  additional  expenditures  during  the 
RDTSE  program  phases.  An  attempt  has  been  made  in  Figure  5  to  show  why  this  is 
not  necessarily  so,  and  to  show  that  real  cost  reductions  can  occur  during  the 
RDTSE  phases  if  all  of  the  "new  look"  tenets  are  practiced  at  all  levels  of 
system  assembly.  Figure  5  shows  that  significant  decreases  in  RDTSE  costs 
should  accrue  by  the  time  the  contractor's  systems  tests  in  the  field  are 
concluded,  because  removal  of  failures  at  lower  assembly  levels  result  in  a 
reduced  number  of  field  tests  which  are  much  more  expensive  than  laboratory 
tests.  For  example,  repeating  one  $20,000  flight  test  because  of  a  railed  com¬ 
ponent  could  easily  fund  laboratory  reliability  growth  testing  for  several 
months.  "New  look"  methodology  should  result  in  a  substantial  reduction  of  the 
cost  of  ownership  as  shown  by  the  upper  curve.  One  reason  fo>"  this  reduction 
is  that  the  large  expense  of  performing  engineering  investigations  and  correc¬ 
tive  actions  during  operational  deployment  are  replaced  by  less  expensive  tests 
at  the  contractor's  facilities  in  the  laboratory  prior  to  delivery. 


Figure  5  is  conceptual  in  nature  and  could  benefit  by  substantiation  with 
facts  that  are  traceable  to  existing  programs.  However,  on  the  basis  of  the 
information  presented,  it  is  thought  that  people  who  propose  conversion  to  “new 
look"  methodology  should  assimilate  the  concepts  that  are  reflected  in  Figure  5, 
and  attempt  to  sell  the  "new  look"  methodology  to  procuring  activities  on  that 
basis.  The  alternative,  which  may  prove  quicker  and  more  efficient  in  the  long 
'un,  is  to  institutionalize  the  methodology  within  the  military  standards  and 
specifications  that  are  utilized  in  procurement,  the  danger  in  the  institutiona¬ 
lization,  to  be  avoided  if  possible,  is  the  conflict  with  tailorability. 


CONCLUSIONS  ANO  RECOFWENDATIONS 


While  the  concept  that  a  new  discipline  called  Environmental  Engineering  is 
desirable  has  been  presented  in  this  paper,  such  a  defined  discipline  does  not 
exist.  It  is  thought  that  published  guidelines  for  practice  of  the  function 
would  be  most  helpful  toward  achieving  objectives  of  readiness,  reliability,  and 
availability  of  field  hardware.  The  IES  has  such  a  document  entitled 
"Environmental  Engineering  Management"  in  preparation,  and  expects  to  release  it 
in  May  1982.  A  recommended  practices  document  entitled  "Environmental  Test 
Program  Management"  (Reference  2)  should  be  released  in  September  1981.  It  is 
probable  that  the  IES  will  continue  to  encourage  the  initiation  of  a  specialized 
discipline  aimed  at  the  objectives  described  above  in  colleges,  government  and 
industry.  The  question  of  whether  or  not  an  Environmental  Engineering  Program 
Management  standard,  similar  to  MIL-STD-78S,  should  be  issued  is  still  trouble- 
seme,  primarily  from  the  standpoint  of  the  conflict  between  institutionalization 
and  tailoring.  However,  Invocation  of  the  necessary  environmental  engineering 
tasks  early  in  hardware  development  programs  is  essential  for  the  effective 
achievement  of  reliability  through  design,  rather  than  assessment  of  reliability 
througn  tests. 

Current  recommendations  with  respect  to  the  environmental  engineering  func¬ 
tions  may  be  suronarized  as  follows: 
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1.  Procurement  documents  should  reflect  guidelines  for  program 
environmental  engineering.  These  could  take  the  form  of  Industry 
standards  such  as  the  Environmental  Engineering  Management  Practice 
to  be  published  by  the  IES  (Reference  2).  Another  form  could  be  a 
new  Department  of  Defense  pamphlet  or  other  guideline-document 
which  lists  environmental  engineering  requirements  and  tasks. 

Those  task  descriptions  might  be  similar  to  the  descriptions  In  the 
current  MIL-STD-785B. 

2.  Procuring  agency  enforcement  of  necessary  front-end  environmental 
emphasis  should  be  accomplished  via  preparation  of  specific  reports 
by  contractors  for  approval  by  the  procurement  agency  prior  to  pro¬ 
ceeding  at  such  points  in  the  program  as  the  following: 

a.  Environmental  Program  Plan— prior  to  validation. 

b.  Environmental  Profile  Report-early  In  validation. 

c.  Environmental  Design  and  Test  Criteria  Report— 
during  validation. 

d.  Environmental  Test  Plans  and  Report— during  full 
scale  development. 

e.  Operational  Environmental  Validation  Measurement 
Report— prior  to  DSARC  I II A. 

3.  Procurement  documents  such  as  MIL-STDs  785,  781  and  810  should 
enumerate  certain  environmental  functions  as  being  necessary 
prerequisites  to  testing  (such  as  the  functions  which  have  been 
proposed  for  Inclusion  in  MIL-STD-810D) ,  and  which  are  listed 
in  Table  II. 

4  Procurement  agencies  should  seriously  review  the  timeliness  of 
environmental  engineering  tasks  and  environmental  tests,  which 
should  cause  them  to  give  serious  consideration  to  specifying  cer¬ 
tain  reliability  development  tests  during  the  validation  phase, 
even  though  such  tests  may  be  duplicated  by  competing  contractors. 
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Determine  LCC 


Figure  1:  Effects  of  Early  Program 
Decisions  on  Life-Cycle-Costs 
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SERVE  AS  POINT  OF  ACCOUNTABILITY  FOR  ENVIRONMENTS 

INTEGRATE  ENVIRONMENTAL  EFFORTS  THROUGHOUT  RDT&E 
PROGRAM  PHASES 
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Table  III:  Environmental  Engineering 
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Reduction  in  field  reliability 
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Figure  4:  Reliability  Improvement  Trend 
with  Environmental  Testing 
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Tabte  V:  Testing  Program  Summary 
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Figure  5:  TAAF  Impact  on  Maturity 
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3.3 


BACKGROUND 


Avionics  equipment  reliability  has  been  the  subject  of  many  broad 
investigations  within  the  Department  of  Defense  (Refs.  1,2, 3, 4).  In  general, 
the  level  of  deployed  system  reliability  has  been  less  than  anticipated  by  the 
acquisition  agency,  desired  by  the  logistician  and  operational  user. 
Additionally,  a  significant  measure  of  this  disparity  has  been  attributed  to 
the  lack  of  correlation  between  the  laboratory  and  field  environmental  stress 
conditions.  One  study  showed  that,  on  the  average,  over  50  percent  of  all 
field  failures  of  avionics  subsystems  were  due  to  the  effects  of  temperature, 
humidity,  altitude  and  vibration  (Ref.  5)  (Figure  1).  However,  for  any  given 
subsystem,  the  percentages  of  failures  caused  by  each  environmental  stress  can 
strongly  diverge  from  this  average  value.  For  example,  humidity  has  been 
shown  to  have  caused  60  percent  of  the  field  failures  of  one  avionics  subsystem 
(Ref.  6),  while  the  direct  effects  of  altitude  have  caused  up  to  ten  percent  of 
the  field  failures  of  another  subsystem  (Ref.  5). 


VIBRATION  >4% 


Figure  1.  Typical  Distribution  of  Field  Failures 
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BASIS  FOR  COMBINED  ENVIRONMENT  RELIABILITY  TEST  (CERT) 


Analysis  of  environmental  data  shows  that  environmental  conditions  of  tem¬ 
perature,  humidity,  altitude,  and  vibration  vary  significantly  throughout  the 
mission  of  an  aircraft.  In  fact,  strong  correlation  exists  between  the  mission 
of  an  aircraft  and  the  environmental  stress  combinations  that  are  impressed  upon 
its  avionics  at  any  given  instant  of  time  throughout  its  flight  (Ref.  5).  (An 
aircraft  performing  a  combat  mission  Imposes  significantly  different  environmen¬ 
tal  stresses  upon  its  avionics  than  if  it  had  been  performing  a  cross-country 
ferry  mission).  This  correlation  between  the  aircraft  flight  conditions  and  the 
environment  under  which  its  avionics  must  function,  suggests  that  the  environ¬ 
mental  combinations  should  be  time- sequenced  in  the  test  criteria  similar  to  how 
the  aircraft  is  flown  so  that  the  resultant  laboratory  test  conditions  would  be 
more  representati ve  of  the  field  environment.  This  concept  has  been  called  com¬ 
bined  environment  mission  profile  testing.  This  concept  forms  the  basis  for  the 
CERT  (combined  environment  reliability  testing)  approach  evaluated  in  this 
study. 


C0M8INED  ENVIRONMENT  MISSION  PROFILE 


Combinations  of  environmental  conditions  characteristic  of  those  that  occur 
during  aircraft  ground  park,  takeoff,  cruise,  combat,  and  other  major  aircraft 
flight  conditions  are  put  together  into  a  dynamic  test  sequence  that  follows  the 
order  in  which  they  would  occur  during  typical  aircraft  missions.  Thus,  the 
behavior  of  the  equipment  in  the  laboratory  test  should  closely  approach  its 
performance. 


INITIAL  TESTS 


Mr.  Prather  and  Mr.  Earls  (Ref.  7)  evaluated  a  combined  environments 
mission  test  profile  that  included  all  reasonable  environmental  conditions  for 
the  test  item,  a  fighter  aircraft  radar  system.  The  missions  and  their  mix  for 
the  aircraft  that  carried  this  equipment  was  generalized  In  terms  of  percent  of 
aircraft  life  spent  in  each  major  mission  phase  (ground  park,  takoff,  climb. 
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cruise,  combat,  etc.).  The  test  profile  was  formulated  to  maintain  a  comparable 
percent  of  mission  phase  similarity  to  the  generalized  mission.  It  was  found 
for  this  aircraft  that  this  similarity  could  be  achieved  by  using  two  design 
missions,  a  training  and  a  functional  checkout  mission.  The  test  profile  con¬ 
sisted  of  alternating  between  the  training  and  the  functional  checkout  missions 
with  a  climatic  environmental  soak  between  each  mission.  The  climatic  soak  con¬ 
ditions  alternated  between  tropic  and  arctic  and  changed  after  every  second 
mission. 

Upon  using  this  profile,  Mr.  Prather  and  Mr.  Earls  found  close  agreement 
between  the  laboratory  and  field  reported  MTBF  (mean  time  between  failures). 

The  agreement  occurred  when  the  definition  of  what  constituted  a  failure  was 
made  consistent  with  field  practices. 

The  Initial  test  of  the  CERT  concept  created  enough  Interest  to  plan  and 
conduct  a  large-scale  comprehensive  program  to  evaluate  the  CERT  methodology. 

The  program  was  jointly  accomplished  with  Aeronautical  Systems  Division's  PRAM 
Program  Office,  the  ASD  Deputy  of  Engineering,  and  the  Air  Force  Wright 
Aeronautical  Laboratories  ( AFWAL ) . 

OVERALL  APPROACH 


The  basic  objective  of  the  CERT  Evaluation  Program  was  to  determine  the 
most  cost  effective  formulation  of  CERT  testing  for  the  early  identification  of 
deficiencies  and  for  providing  insight  into  how  the  equipment  will  perform  in 
operational  service.  In  order  to  achieve  this  objective,  a  comparison  was  made 
of  CERT  results  versus  MIL-STD-781  for  Identifying  significant  environmentally- 
induced  field  failure  modes  In  the  laboratory. 

The  degree  of  correlation  between  CERT  failure  rates  and  field  failure 
rates  was  also  determined.  This  was  accomplished  by  selecting  test  samples 
broadly  representatl ve  of  the  avionics  population  In  regular  use  in  operational 
aircraft.  These  test  samples,  either  multiple  or  single  Line  Replaceable  Unit 
(LRU)  systems,  were  subjected  to  mission  profile  combined  environments 
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representative  of  their  location  in  the  aircraft.  They  were  operated  as  in 
actual  service  and  monitored  for  performance  and  operating  characteristics 
throughout  the  test  insofar  as  practical.  Out  of  specification  performance  was 
noted,  as  well  as  component  failures.  A  detailed  failure  analysis  was  conducted 
on  some  but  not  all  failed  components  in  an  attempt  to  identify  the  cause  of 
failure. 

The  work  effort  also  included  determination  of  facility  capabilities  and 
most  cost  effective  facility  alternatives,  along  with  recommended  appropriate 
changes  In  reliability  test  methods. 

SELECTION  OF  TEST  SPECIMEN  CANDIDATES 


Selection  of  avionics  equipment  for  CERT  evaluation  included  a  broad  range 
of  considerations.  It  was  found  necessary  to  categorize  types  of  aircraft, 
classes  of  avionics,  determine  number  of  subsystems  to  be  tested,  and  addi¬ 
tional  detailed  selection  criteria  to  prioritize  the  effort  according  to 
rational  methodology. 

Categories  of  aircraft  include  fighter,  bomber,  cargo  and  trainer.  These 
aircraft  usually  have  different  avionics  subsystems,  are  used  in  a  significantly 
different  manner,  thereby  having  significantly  different  mission  profiles  and 
consequently  different  environmental  conditions. 

Categories  of  avionics  investigated  include  communications/navigations/ 
identification,  control  and  displays,  intertial  navigation,  flight  instruments. 
Electronic  Counter-Measure  (ECM),  electronic  flight  controls,  radars,  computers 
and  electro-optical  subsystems.  Selecting  equipment  from  this  broad  base  was 
expected  to  yield  results  that  would  effectively  evaluate  the  CERT  concept  for  a 
wide  variety  of  avionic  applications. 

Additional  selection  criteria  that  were  very  pertinent  to  determining 
equipment  to  be  tested  Included  availability  of  MIL-STD-781  test  data,  as  this 
requirement  was  necessary  for  a  baseline  comparison  of  the  degree  of  effec- 
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tlveness  of  new  test  procedures  compared  to  present  or  past  test  methods.  Low 
field  MTBF  equipment  is  one  criterion  which  was  utilized  to  provide  a  failure 
data  base  in  a  cost  effective  test  time.  Extensive  recent  field  experience  and 
recent  vintage  equipment  was  also  desirable.  The  field  experience  provides 
extensive  failure  data  under  operational  usage  for  correlation  comparison  of  new 
laboratory  test  Induced  failures.  Recent  vintage  equipment  help  to  Insure  that 
test  results  were  applicable  to  state-of-the-art  designs  and  Increased  the  con¬ 
fidence  In  obtaining  similar  results  for  new  equipment.  Additional  test  selec¬ 
tion  criteria,  established  to  meet  program  schedule  restraints  and  objectives 
included  availability  of  spares  and  replacement  units,  Aerospace  Ground 
Equipment  (AGE)  or  mockup  availability  supplemental  conditioning  (cooling)  para¬ 
meters,  size  of  equipment  tested  and  test  chamber  availability. 


QUANTITY  OF  SUBSYSTEMS  TESTED 


Combinations  of  aircraft  performance  and  type  of  equipment  cooling  were 
considered  in  determining  the  number  of  subsystems  to  be  tested.  Aircraft  were 
categorized  as  high  performance  and  low  performance,  with  either  supplemental 
air  (environmental  control  system  [ECS]  cooling  air)  or  ambient  air.  It  was 
decided  to  concentrate  on  high  performance  aircraft  since  these  appllctions  have 
the  most  significant  differences  in  environmental  stresses. 

For  purposes  of  the  evaluation  program,  it  was  determined  that  a  minimum  oc 
two  specimens  of  each  subsystem  would  be  tested.  This  is  based  on  the  assump¬ 
tion  that  the  MTBFs  of  fielded  systems  followed  a  gaussian  distribution;  and 
that  if  the  MTBF  of  a  selected  subsystem  was  within  one  standard  deviation  of 
the  mean  of  all  fielded  subsystem's  MTBFs,  It  was  representative  of  fielded  sub¬ 
systems.  Testing  two  subsystems  then  ensured  (with  90  percent  confidence)  that 
a  subsystem  representati ve  of  the  field  population  was  being  tested. 
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DURATION  OF  EACH  TEST 


A  test  length  of  15  times  the  equipment  field  MTBF  or  three  calendar  months 
was  established  to  obtain  engineering  confidence  and  to  reflect  what  amount  and 
duration  of  testing  would  generally  be  acceptable  in  an  acquisition  program. 

NUMBER  OF  TESTS 

Each  equipment  system  was  exposed  to  up  to  three  different  test  sequences. 
These  test  sequences  were  called  CERT,  CFRT  Without  Altitude  and  MI L -STD-731 C 
Appendix  B;  and  were  labeled  CERT  I,  II  and  III,  respectively. 

TEST  DATA  GENERATED 

A  total  of  six  different  equipment  types  were  used  in  the  CERT  Evaluation 
Program.  These  equipment  tests  are  documented  in  References  8  through  13. 
Typically,  for  each  test  sequence  randomly  selected,  test  Items  were  obtained 
from  depot  repair  lines.  Generally,  different  serial  numbered  units  were  used 
in  each  test  sequence.  The  test  items  were  bench  checked  before  start  of  the 
respective  test  sequence.  During  environmental  testing,  each  test  item  was 
electrical ’y  active  and,  in  most  cases,  actually  operating  and  performing  its 
function  as  opposed  to  just  running  bit  tests  or  being  operated  periodically 
•-'■:-i';g  the  test  cycle. 

Table  I  lists  the  equipment  tested  in  the  program,  the  aircraft  application 
simulated,  total  number  of  equipment  on-hours  under  environmental  test,  and 
number  of  different  serial  numbered  units  used  in  each  test  sequence. 

Table  II  summarizes  the  number  of  failures  that  occurred  during  environmen¬ 
tal  simulation  for  each  test  sequence.  The  failures  that  occurred  were 
classlf’eo  as:  cann  t  duplicate  (CND),  hard,  or  adjustment.  A  CND  failure  is 
when  the  equipment  is  taken  out  of  the  test  and  checked  out  In  the  normal  bench 
environment  and  no  failure  could  be  found.  However,  the  failure  generally  could 
be  repeated  at  will  by  applying  the  appropriate  environmental  stress  conditions 
in  the  test  profile  Hard  failures  are  failures  which  are  still  present  when 
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the  equipment  is  removed  from  the  test  and  checked  out  in  the  normal  bench 
environment.  Typically,  a  part  had  to  be  replaced  to  correct  these  failures. 
Adjustment  type  failures  were  categorized  as  occurring  when  the  performance  of 
the  item  degraded  below  the  failure  threshold;  however.  It  could  be  returned  to 
required  performance  levels  by  adjusting  the  unit  appropriately. 


TABLE  I.  SUMMARY  OF  CERT  EVALUATION  PROGRAM 


CERT 

I 

CERT  II 

mam 

EQUIPMENT 

A/C 

APPL 

TOTAL 

ON  HRS 

#  OF 

CYS  USED 

TOTAL 
ON  HRS 

1» 

EEBBa? 

TOTAL 

ON  HRS 

*  OF  j 

CYS  1 : c  t U 

AN/ARC-164 

A-7D 

1913 

3 

- 

- 

- 

- 

RT-868A/APX-76 

F— 15 

2082 

6 

- 

- 

851 

6 

CN-1260/ASN-90 

A-7D 

705 

3 

- 

- 

675 

3 

AN/ARC-109 

F-111F 

1880 

6 

1135 

5 

1392 

5  I 

RT-1063B/APX-101 

F— 15 

1008 

3 

1009 

3 

930 

4 

AN/ARN-84 

F-5 

1122 

5 

1099 

5 

1122 

5 

CERT  r  All  environments  of  the  mission  profile  are  developed  from  either 

actual  field  data  or  by  use  of  computerized  models  based  on  expected 
aircraft  parameters.  The  environments  Include  altitude,  temperature, 
rate  of  change  of  temperature,  humidity,  vibration  and  Input  voltage. 

CERT  II:  The  same  as  CERT  I  except  that  altitude  is  omitted. 

CERT  III:  The  same  as  CERT  II  except  that  the  environmental  profiles  are  In  a 
accordance  dance  with  the  standardized  profiles  of  MIL-STD-781C. 
Appendix  B,  Section  50.4  entitled  Combined  Environments  for  Jet 
Aircraft  Equipment. 
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TABLE  II.  TEST  FAILURES  BY  CATEGORY 


CERT 

II 

CERT  III 

CERT  III 

EQUIPMENT 

A/C 

APPL 

HARD 

CND 

ADJ 

HARD 

CND 

ADJ 

HARD 

CND 

ADJ 

AN/ARC-164 

A-7D 

13 

5 

9 

- 

- 

- 

- 

- 

- 

RT-868A/APX-76 

F-15 

18 

1 

2 

- 

- 

- 

9 

4 

1 

CN-1260/ASN-90 

A-7D 

0 

7 

0 

- 

- 

- 

- 

2 

- 

AN /ARC -109 

F-111F 

9 

9 

4 

3 

3 

5 

1 

2 

1 

R1-1063B/APX-101 

F-15 

2 

2 

0 

4 

1 

- 

14 

8 

0 

AN/ARN-84 

F-5 

7 

5 

0 

6 

2 

1 

6 

10 

0 

ANALYSIS  OF  TEST  DATA  AND  DISCUSSION 


The  test  data  from  the  CERT  Evaluation  Program  were  analyzed  to  determine 
the  most  cost  effective  formulation  of  CFRT  for  the  early  identification  of 
>.<f1c  it’k  ies  and  .'or  providing  insight  into  how  an  equipment  item  will  perform 
iii  operar iona1  service. 

Figure  2  shows  that  the  majority  of  the  modes  of  failure  occurred  after  a 
few  hundred  hours  of  testing.  The  curve  shown  in  this  figure  was  a  spline  fit 
of  the  curve  Y=AX®.  Assuming  that  the  occurrence  of  a  new  failure  mode  follows 
distribution,  then  90  percent  of  the  failure  modes  occurred  within 
\3  ,Q7  or  36r  on  ho"rs  for  CERT  I,  II  and  III,  respectively. 

An  Important  consideration  was  the  percentage  of  the  test  failure  modes 
a*  •.< .  u’ red  in  CCK T  » .1  t  wore  field  failure  modes.  The  fewer  test  peculiar 
failure  modes,  the  hiyher  the  'evel  of  engineering  confidence  that  a  test  Iden- 
fifled  deficiency  else  wouid  occur  in  the  field.  All  CERT  failure  modes  did  in 
,n  r  00  ur  in  the  fluid.  Tht-re  were  no  test  peculiar  failure  modes. 


On  Hours  Accumulated  by  Failed  Item 


(a)  CERT  I 


Figure  2.  Occurrence  of  Distinct  Failure  Modes 
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The  significance  of  this  result  was  somewhat  mute  because  sooner  or  later 
everything  will  fail  in  the  field.  A  more  meaningful  way  to  address  this  issue 
was  to  determine  if  the  test-identified  failure  modes  occur  in  significant 
quantities  in  the  field.  These  modes  of  failure  occur  in  such  large  quantities 
that  they  may  drive  logistic  costs. 

In  general.  Air  Force  field  data  D056  do  not  indicate  the  actual  repair  tr 
the  part  level  so  that  a  direct  comparison  on  failure  modes  was  not  possible 
(Ref.  14).  Therefore,  it  was  decided  to  use  an  Air  Force  field  maintenance 
coding  system.  This  code  is  called  "How  Malfunctioned"  (How  Mai),  codes  which 
describe  why  the  maintenance  was  necessary.  It  is  recognized  that  these  codes 
were  somewhat  subjective  since  they  depend  upon  the  judgement  of  the  persons 
recording  the  failure.  A  comparison  between  test  and  field  data  was 
accomplished  and  the  results  are  shown  in  Table  III. 

TABLE  III.  AVERAGE  PERCENT  OF  FIELD  FAILURE  HOW  MAL  CODES  OBSERVED  IN  TEST 


CERT 

MEAN 

STANDARD  DEVIATION 

I 

45.6 

28.7 

II 

31.5 

36.8 

III 

39.4 

38.4 

From  Table  III,  it  appears  that  CERT  I  is  better,  but  there  was  no  statistical 
reason  to  say  that  the  three  CERTs  are  not  equally  effective.  To  put  these  ;v 
centages  into  perspective,  consider  the  data  shown  in  Figure  1.  This  figure 
Indicates  that  about  54  percent  of  field  failures  on  the  average  are  due  to  fac 
tors  or  environmental  stresses  not  present  in  the  CERT  tests.  Therefore,  it 
seems  reasonable  that,  on  the  average,  the  best  that  can  be  expected  from  a 
CERT  is  that  it  shows  46  percent  of  the  field  failure  modes,  or  field  How  Mai 
codes.  Therefore,  CERT  I  would  be  identified  as  (45.7/46)  percent  of  antici¬ 
pated  quantity  of  How  Malfunction  codes.  On  this  basis,  the  three  CERT  tests 
(I,  II  and  III)  were,  respectively,  99,  68  and  86  percent  effective  in  iden¬ 
tifying  the  anticipated  quantity  of  field  How  Malfunction  codes. 
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MTBF 


Another  way  of  comparing  test  effectiveness  was  to  compare  test  MTBF  to 
field  MTBF.  The  limitation  of  using  normal  Air  Force  0056  system  data  (Ref.  15) 
was  well  recognized.  Even  so,  there  was  no  attempt  to  screen  or  adjust  these 
data  to  compensate  for  the  fact  that  many  sources  such  as  personnel  errors  were 
not  included  in  the  CERT.  This  approach  was  selected  because  the  reported 
field  MTBF  was  the  most  widely  used  means  of  evaluating  how  an  equipment  per¬ 
forms  in  operational  service. 

Table  IV  lists  the  field  MTBF  values  for  these  equipment  items  for  a 
12-month  time  period  that  Included  the  period  of  time  that  the  equipment  was 
used  in  the  CERT  Evaluation  Program.  This  helped  to  increase  the  likelihood  of 
a  consistent  configuration  for  the  item  used  in  the  program  and  deployed. 

Field  data  systems  would  only  count  two  of  the  three  failure  categories 
shown  in  Table  II  as  being  field  failures.  These  two  categories  are  Hard  and 
Adjustment  (Ref.  16)  since  CND  laboratory  failures  are  not  counted  as  failures, 
but  as  maintenance  actions.  Using  the  data  shown  in  Tables  I  and  II,  the  MTBF 
values  in  Table  IV  were  calculated.  Table  IV  also  includes  initial 
MIL-STD-781B  test  values  for  these  equipment  items. 


TABLE  IV.  MTBF  VALUES 


— 

EQUIPMENT 

A/C  APP 

— mr 

FIELD 

CERT  I 

CERT  II 

CERT  III 

MIL-STD-781B 

AN /ARC -164 

A-7D 

44 

87 

- 

- 

1000 

RT868A/APX-76 

F-15 

133 

104 

- 

85 

1748 

CN-1260/ASN-90 

A-7D 

156 

★ 

- 

★ 

1885 

AN/ARC-109 

F-111F 

95 

145 

142 

969 

619 

RT - 1063B/APX-101 

F-15 

225 

504 

252 

66 

762 

AN  'ARM -84 

i _ 

F-5 

_ 

170 

_ 

160 

157 

187 

670 

-1 


1 

I 


To  interpret  and  identify  data  trends,  the  data  were  normalized  by  deter¬ 
mining  test-to-field  MTBF  ratios,  and  calculating  confidence  intervals  using 
small  sample  size  statistics,  Student  t  distribution.  For  this  calculation,  it 
will  be  assumed  that  the  MTBF  ratios  came  from  a  population  that  is  log  nor¬ 
mally  distributed  (Ref.  17).  For  variables  that  can  only  be  positive,  MTBF 
ratios,  the  log  normal  distribution  gives  a  more  logical  fit  since  it  matches 
the  physical  realities  that  MTBF  ratio  cannot  be  negative.  The  normal 
distribution  extends  from  minus  to  plus  infinity,  while  log  normal  extends  from 
zero  to  plus  infinity.  Additionally,  a  Chi  Square  goodness  of  fit  test  shows 
that  the  log  normal  distribution  is  a  good  fit  to  these  data.  Figure  3  and 
Table  V  show  the  results  of  these  calculations. 


Figure  3.  90%  Confidence  Intervals  for  MTBF  Ratio 


TABLE  V.  90%  CONFIDENCE  BANDS  ON  TEST-TO-FIELD  MTBF  RATIOS 


LOWER 

UPPER 

CERT  I 

0.94 

2.2 

CERT  II 

0.70 

1.9 

CERT  III 

0.17 

7.1 

MIL-STD-781B 

4.20 

16.4 
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’*'is  analysis  showed  that  there  was  not  a  single  fixed  proportionality 
constant  Detween  test  and  field  MTBF  values.  This  appears  reasonable  since 
field  data  include  a  significant  level  of  failures  caused  by  factors  not 
i  .,'Uded  in  the  CERT.  Also,  the  CERT  mission  profiles  represented  nominal  or 
average  flight  conditions  flown  by  pilots. 

An  analysis  of  these  test  results  on  a  purely  statistical  basis  suggests 
that  there  is  no  difference  among  the  three  tests.  It  is  felt  by  the  author 
that  a  quotation  from  Mr.  Aaron  Levenstein  is  appropriate: 

"Statistics  are  like  a  bikini.  What  they  reveal  is  suggestive. 

But  what  they  conceal  is  vital". 

in  '.Ms  case,  what  is  concealed  is  tne  fact  that  CERT  I  and  CERT  II  gave 
more  consistent  correlation  between  test  and  field  results.  Factors  which  tend 
to  muddle  a  purely  statistical  analysis  of  this  data  are  also  present  in  any 
equipment  acquisition  program.  These  factors  are:  small  number  of  test  items, 
short,  test  duration,  and  high  level  of  failures  induced  by  factors  other  than 
env i runmental  stresses.  In  any  field  data,  therefore,  reliability  data  should 
ho  •  ~o dprna  emineerino  information. 


NEED  FOR  CERT  DURING  &CQ!J  I  SIT  ION  PROCESS 

CER'  has  several  distinct  advantages  that  should  be  capitalized  during  the 
acquisition  process.  Because  of  the  realistic  nature  and  combinations  of 
environmental  stresses  used  in  the  test,  an  equipment  design  can  be  quickly  eva- 
udLf-i  without  any  engineering  prejudgement  as  to  what  environmental  stresses, 
combinations  of  stresses,  or  sequences  of  stresses  are  the  most  severe  for  the 
specific  item  to  be  tested.  Therefore,  the  test  item  indicates  where  its 
derecti  dei ici<rxics  are  and  what  stress  states  are  most  critical  very 
.it:-  •  :  •  i  f  ’c  p'e judgement. 
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During  the  cost  effectiveness  study  (Ref.  14),  Hughes  Aircraft  Company 
identified  seven  failure  modes  that  were  never  observed  with  any  traditional 
testing  accomplished  on  the  equipment  listed  in  Table  I.  The  occurrence  of 
these  field  failure  modes  during  CERT  may  have  been  due  to  environmental  stress 
synergism,  or  stress  sequence.  In  either  event,  these  seven  field  failure 
modes  were  significant  field  failure  modes  for  these  systems. 

DURATION  OF  TESTING 


The  length  of  the  tests  in  the  CERT  Evaluation  Program  was  relatively 
short  when  compared  to  traditional  reliability  demonstration  tests  and  longer 
than  most  environmental  qualification  tests.  The  duration  of  any  testing 
effort  is  somewhat  related  to  reliability  requirements.  Traditionally, 
because  of  the  overly  optimistic  reliability  predictions  that  are  given  during 
the  acquisition  of  a  system,  the  equipment  user  asked  for  much  more  reliability 
than  he  needed.  For  example,  look  at  Figure  3;  MIL-STD-781B  test  MTBF  values 
are  about  eight  times  greater  than  what  was  achieved  in  deployment.  Therefore, 
an  equipment  user  will  request  more  reliability  than  actually  needed  so  that 
the  delivered  equipment  reliability  may  meet  his  real  requirements.  This  over- 
requesting  of  reliability  requirements  drives  test  costs  because  test  durations 
have  been  traditionally  based  on  requested  reliability  in  order  to  achieve  sta¬ 
tistical  confidence  in  the  demonstrated  MTBF  values. 

Instead  of  attempting  to  fix  the  test  duration  on  the  basis  of  statistical 
confidence  on  MTBF  values,  a  different  approach  seems  to  be  appropriate  based 
on  data  presented  herein.  Consider  Figure  2,  which  shows  that  300  to  400 
operating  hours  under  test  identifies  the  majority  to  field  failure  modes  to  be 
observed  in  the  test.  Any  additional  testing  beyond  this  point  tends  to 
just  repeat  the  already  observed  failure  modes.  A  study  by  Hughes  found  that 
between  300  to  600  operating  hours  per  test  item  are  needed  to  identify  failure 
modes  that  would  not  appear  in  any  traditional  test  (Ref.  14).  A  total  of  1300 
operating  hours  among  four  test  items  would  be  needed  to  have  high  confidence 
that  a  unique  mode  of  failure,  if  possible,  could  occur. 
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COST  EFFECTIVENESS 


The  cost  effectiveness  of  doing  CERT  was  studied  by  Hughes  Aircraft  (Ref. 
14).  The  study  approach  was  to  quantify  the  cost  effectiveness.  To  accomplish 
this  study,  Hughes  reviewed  all  the  data  collected  on  the  six  equipments  shown 
in  Table  I.  The  failures  that  occurred  in  the  CERT  program  were  screened  to 
determine  which  failures  could  be  considered  as  being  "correctable"  failures. 

To  be  classified  as  correctable,  the  failure  had  to  meet  the  following  criteria 

a.  Can  reasonably  be  caused  by  the  field  environment. 

b.  Can  be  expected  to  occur  in  all  or  at  least  a  significant 
proportion  of  the  serial  numbers  of  the  equipment. 

c.  Could  have  been  eliminated  from  future  occurrence  by  a  design 
change  which  was  technologically  possible  at  the  time  of  the 
original  development  of  the  equipment. 

d.  Would  have  required  the  use  of  mission  profile  testing  to  be 
detected;  i.e.,  would  not  be  revealed  by  more  traditional 
test  program. 

Because  of  the  restrictive  nature  of  these  criteria,  only  7  failure  modes 
were  considered  in  the  costing  analysis.  These  criteria  were  appropriate  for  a 
quantified  cost  study  but,  as  recognized  by  Hughes,  the  cost  savings  benefits 
estimated  by  this  approach  will  be  significantly  understated. 

The  cost  effectiveness  analysis  consisted  of  determining  the  cost  of  con¬ 
ducting  CERT,  making  design  changes,  and  Implementing  these  design  changes  in 
the  production  line.  These  costs  were  compared  to  the  logistic  maintenance 
•d’Oi  and  spare  part  savings  to  be  accrued  if  these  correctable  failures  did 
■'•■■t  occur  in  the  field.  It  was  found  that  for  these  six  equipments,  these 
costs  were  recovered  within  two  years  through  logistic  maintenance  cost 
savings. 
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The  equipment  used  in  the  CERT  Evaluation  Program  were  systems  that  were 
generally  more  mature  than  a  new  product  just  being  produced.  A  less  mature 
system  will  have  more  defects  and  deficiencies  that  will  tend  to  make  CERT 
appear  even  more  cost  effective. 

Another  major  cost  effectiveness  consideration  of  CERT  is  suggested  by 
Figure  3.  The  amount  of  test  time  needed  to  test  an  equipment  can  be  reduced 
if  the  equipment  user  and  acquisition  agency  make  use  of  the  lower  ratios  bet¬ 
ween  test  and  field  MTBF  that  exist  for  CERT.  Instead  of  asking  for  a  1000  hour 
MTBF  when  only  100  is  desired,  significant  cost  savings  may  be  realized  in  the 
design  process,  use  of  appropriate  parts  quality,  and  test  durations. 

CONCLUSIONS 


Two  sets  of  conclusions  are  presented  because  of  a  difference  in  viewpoint  as 
to  how  to  interpret  the  data.  Both  viewpoints  are  presented  because  it  is  felt 
that  each  perspective  is  useful  to  understand  the  CERT  program  results. 

The  nine  following  conclusions  were  reached  by  Hughes  Aircraft  In  their 
CERT  evaluation  program: 

1.  The  additional  costs  associated  with  CERT  I,  1.  e.,  with  altitude, 
are  not  justified. 

2.  The  cost-benefits  of  CERT  testing  during  development/early 
production  will  probably  exceed  the  values  determined  in  this  study 
due  to:  a)  a  large  number  of  correctable  failures  can  be 
expected;  b)  any  RTOK's  uncovered  and  corrected  will  Increase  the 
potential  field  savings. 

3.  On  the  average,  at  least  one  correctable  failure  will  be  found 
in  a  properly  conducted  Mission  Profile  Test.  This  will  pay  for 
Itself  in  between  2  and  4  years,  given  that  a  substantial  inventory 
Is  planned. 

4.  The  above  third  conclusion  is  unlikely  to  be  valid  unless  a  minimum  of 
four  test  samples  are  employed  for  at  least  1300  total  hours  of 

test,  with  approximately  equal  hours  on  each  sample. 
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5.  The  potential  cost  benefits  of  CERT  testing  can  be  very  easily  dissi¬ 
pated  by  a  number  of  factors  extraneous  to  the  technical  merits 

of  the  test.  Most  Importantly,  over-zealous  corrective  action 
on  isolated  failures  which  are  not  field-related  and  truly  correctable 
will  rapidly  negate  the  potential  benefits. 

6.  The  prior  conclusions  should  not  be  taken  to  Infer  that  CERT  testing 
will  significantly  improved  field  reliability.  The  proportion  of  the 
total  field  failures  (or  CERT  test  incidents)  which  were  correctable 
was  generally  small.  Thus  only  a  small  increase  In  field  reliability 
should  be  expected. 

7.  The  ability  of  Mission  Profile  Testing  to  replicate  field  failure 
rates  and  modes  is  marginal  on  mature  equipment.  The  results 
from  testing  development/early  production  equipment  will  probably 
be  even  less  representative  of  what  can  be  expected  to  occur  in 
field  use.  On  the  other  hand,  it  is  undoubtedly  the  best  available 
test  technique,  providing  its  lack  of  precision  Is  recognized. 

8.  In  order  to  assess  the  potential  cost  benefit  of  Mission  Profile 
Testing,  It  was  necessary  to  consider  not  only  when  such  testing  is 
appropriate  but  also  when  It  would  be  inappropriate  and,  therefore 
uneconomic.  Based  on  the  analyses  of  failure  data  and  failure  rates 
described  herein,  and  given  the  basic  premise  that  economical 
screening  must  be  a  highly  accelerated  test,  it  is  concluded  that 
Mission  Profile  Test  conditions  are  totally  Inappropriate  as  the  con¬ 
ditions  for  environmental  stress  screening. 

9.  If  a  single  conclusion  from  this  study  can  be  attempted,  it  is  that 
judicious  use  of  Mission  Profile  Testing  will,  on  average,  be 
beneficial  -  but  It  Is  not  a  panacea. 

The  following  conclusions  were  reached  by  the  author  in  the  CERT 
valuation  Program: 
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1.  CERT  has  been  found  to  be  cost  effective.  On  the  average,  payback  of 
the  costs  associated  with  conduct  of  CERT  is  returned  in  less  than  two 
years  of  field  deployment  because  of  reduced  maintenance  costs. 

2.  CERT  should  be  used  either  In  late  development  and/or  early  production 
when  there  Is  a  mature,  somewhat  stable  design.  A  minimum  of  four  test 
Items,  each  of  which  Is  exposed  to  a  minimum  of  300  operating  on  hours 
under  environmental  stress,  is  recommended. 

3.  There  are  no  significant  differences  between  CERT  I,  II  and  III  for  the 
Identification  of  failure  modes.  Thus,  the  Inclusion  of  altitude  in 
the  test  Is  not  justified. 

4.  The  more  realistic  the  environmental  stress  conditions  used  in  CERT, 
the  more  consistent  are  the  test-to-field  MTBF  ratios.  Specifically, 
the  90  percent  confidence  branch  on  test-to-field  MTBF  ratios  were  0.94 
to  2.2,  0.7  to  1.9  and  0.17  to  7.1  for  CERT  I,  II  and  III, 
respectively. 

5.  CERT  identified  failure  modes  that  occur  in  the  field  but  were  never 
observed  In  any  traditional  single  environment  or  reliability  test. 

6.  Reliability  requirements  need  to  be  more  realistically  stated  so  that 
the  user  who  wants  a  50  hour  MTBF  does  not  ask  for  500  hours.  Such 
over-specification  can  be  eliminated  because  of  the  more  realistic 
nature  of  CERT  test  MTBF  estimates.  Significant  cost  benefits  In  add- 
tion  to  those  Identified  by  Hughes,  can  be  possible  from  proper  state¬ 
ment  of  reliability  requirements. 

RECOMMENDATIONS 

1.  CERT  should  be  used  late  in  equipment  development  and/or  early  in  pro¬ 
duction. 

2.  Reliability  requirements  need  to  be  stated  realistically  by  the  equip¬ 
ment  user.  The  ten  to  one  test-to-field  inconsistency  of  previous 
reliability  demonstration  testing  no  longer  exists. 

3.  CERT  Mission  Profile  Testing  should  be  used  for  failure  mode  Iden¬ 
tification  testing  as  well  as  tests  for  estimating  field  failure  rates. 

4.  CERT  testing  allows  for  the  use  of  fewer  test  Items  In  an  Integrated 
testing  program.  This  Is  achieved  because  the  test  conditions  are 
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realistic  and  do  not  use  up  the  test  Item  like  traditional  testing 
approaches.  Thus,  a  system  can  be  used  interchangeably  In  flight,  per¬ 
formance  and  CERT  tests.  A  short  CERT  test  before  flight  or  perfor¬ 
mance  testing  could  shake  out  design  bugs  that  would  have  delayed  these 
other  tests.  CERT  identifies  these  problems  for  corrective  action 
while  collecting  reliability  data. 

CERT  testing  could  be  used  in  place  of  several  of  the  traditional 
testing  methods.  This  is  recommended  because  CERT  identifies  failure 
modes  that  occur  in  traditional  testing  In  addition  to  field  failure 
modes  not  observed  In  any  traditional  testing.  This  approach  of 
integrated  testing  appears  to  have  additional  cost  benefits  not 
included  in  tne  Hughes  cost  effectiveness  study.  These  cost  savings 
come  from  a  net  reduction  In  the  number  of  tests  to  be  conducted  during 
equipment  acquisition. 
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INTRODUCTION 


The  Pacific  Missile  Test  Center  (PACMISTESTCEN)  has  been  conducting 
acceptance  testing  since  1959  of  guided  missiles  produced  for  the  Navy. 
Initially,  the  acceptance  programs  were  proof  firing  programs  in  which  a 
monthly  sample  of  missiles  were  fired  to  assess  production  quality.  In  the 
early  1970's,  the  increasing  costs  of  missiles  and  the  costs  of  firing 
forced  the  Navy  to  investigate  non-destructive  test  techniques  to  replace 
the  proof  firing  programs.  Following  considerable  research  into  test  tech¬ 
niques,  a  form  of  Combined  Environmental  Reliability  Testing  (CERT)  was 
developed.  A  test  was  designed  for  each  missile  system  that  consisted  of 
subjecting  the  missile  to  combined  environmental  conditions  closely  simu¬ 
lating  the  operating  environments.  As  these  tests  were  developed,  they  w e^e 
used  for  different  purposes.  The  major  applications  are  summarized  in  Table 
I.  The  missile  types  to  which  the  test  were  applied  are  shown  in  Table  II. 

In  the  following  discussion  the  author  will  try  to  pass  on  some  of  the 
lessons  learned  at  PACMISTESTCEN.  This  paper  will  concentrate  on  the  mana¬ 
gement  aspects  of  CERT;  technical  issues  have  been  addressed  elsewhere 
(Refs.  1-7).  Also,  some  unsolved  problems  that  have  arisen  in  CERT  of 
missiles  will  be  mentioned. 
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(1)  These  are  or  were  continuing  tests  conducted  on  monthly  or  quarterly  samples. 

(2)  These  are  planned. 
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DISCUSSION 


There  are  some  lessons  which  are  not  peculiar  to  Combined  Environmental 
Reliability  Test  (CERT),  but  which  were  learned  at  the  Pacific  Missile  Test 
Center  (PACMISTESTCEN)  as  a  consequence  of  applying  CERT.  These  lessons 
would  apply  to  any  reliability  test,  but  they  became  more  important  simply 
because  CERT  was  a  better  test. 

First,  we  became  aware  that  a  reliability  test  Is  only  part  of  a  process. 
The  whole  process  must  Include  analysis  of  failures,  formulation  of  corrective 
action,  and  a  decision  as  whether  the  corrective  action  Is  to  be  taken.  To 
conduct  a  CERT  without  these  elements  is  like  strengthening  one  link  In  a 
chain.  The  other  elements  were  not  so  Important  when  standardized  reliability 
tests  were  run  simply  because  the  results  were  not  expected  to  be  realistic 
anyway.  One  of  the  ways  of  ensuring  closure  of  the  failure  analysis  and 
correction  loop  which  has  worked  well  at  PACMISTESTCEN  Is  the  establishment  of 
a  Failure  Analysis  Review  Board  (FARB).  There  Is  one  FARB  for  each  missile 
type;  It  meets  regularly  and  It  consists  of  representatl ves  from  the  govern¬ 
ment  and  the  manufacturer.  It  Is  a  technical  working  group  vrfilch  reviews  and 
approves  failure  analyses  and  proposed  corrective  actions.  It  does  not  decide 
on  Implementation  of  corrective  actions. 

Second,  another  lesson  which  is  more  general  than  CERT,  is  that  there  is 
an  optimum  level  of  assembly  or  aggregation  at  which  to  test.  If  a  reliabi¬ 
lity  test  is  conducted  at  a  low  level,  say  modules  within  a  missile,  the 
Mean-Time-Between-Fallure  (MTBF)  requirement  will  be  high  and  many  different 
assemblies  must  be  tested.  Consequently,  the  testing  will  be  long  and  costly. 
On  the  other  hand.  If  testing  Is  attempted  at  a  very  high  level,  say  a  complete 
avionics  suite,  the  MTBF  will  be  short  and  the  number  of  functions  to  be 
tested/monitored  will  be  great.  Consequently,  the  test  set-up  will  be  very 
costly  and  It  may  be  Impossible  to  check  all  functions  In  a  time  less  than  the 
MTBF.  In  this  case,  CERT  will  lose  many  of  Its  advantages  relative  to  simply 
putting  the  Item  Into  service  and  observing  failures.  Between  these  two 
extremes  Is  a  range  of  MTBFs  and  functional  complexity  which,  happily  for 
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PACHISTESTCEN,  Includes  air-launched  missiles.  Functional  tests  require  a  few 
minutes  to  an  hour  depending  on  the  thoroughness.  Missile  MTBFs  when 
PACMISTESTCEN  began  CERT  were  generally  less  than  100  hours,  which  seemed 
about  right.  Now  missile  MTBFs  are  In  the  range  of  500  to  1000  hours. 

Because  of  this  higher  MTBF,  missiles  are  now  tested  four  to  six  at  a  time  so 
that  the  aggregate  MTBF  Is  still  around  100  hours.  The  increased  MTBF  com¬ 
bined  with  the  complexity  of  functional  test  equipment  also  led  to  another 
1 esson . 

One  of  the  most  Important  lessons  we  learned  Is  that  a  standard  or  univer¬ 
sal  chamber  design  was  not  possible.  Tailoring  the  test  to  a  specific  missile 
mission  profile  required  a  large  variety  of  temperature  conditioning  equip¬ 
ment,  acoustic  systems  and,  from  a  cost  consideration,  different  volumes  of  the 
very  Important  energy  source,  low  pressure  air.  It  became  Impractical  to  spe¬ 
cify  the  environmental  equipment  until  a  general  review  of  the  potential 
mission  profile  was  completed.  Several  other  factors  Influenced  this  deci¬ 
sion.  Design  and  fabrication  of  the  necessary  Instrumentation  to  energize  and 
monitor  the  missile  was  quite  often  the  most  expensive  and  most  difficult  part 
of  the  facility  development.  The  magnitude  and  complexity  of  the  Instrumen¬ 
tation  normally  did  not  lend  Itself  to  a  temporary  Installation.  One  other 
factor  related  to  dedicated  chambers  was  a  corresponding  Increase  In  missile 
reliability  that  has  occurred  In  the  past  six  years.  The  Initial  missiles 
tested  had  MTBFs  between  50  and  100  hours.  A  reliability  test  could  be 
completed  In  a  relatively  short  time  now,  wf'.h  missiles  having  MTBFs  In  the 
500  to  1000  hour  range,  test  durations  require  chambers  to  operate  around-the- 
clock  most  of  the  time.  There  is  no  opportunity  for  facility  sharing. 

This  around-the-clock  operation  also  taught  us  something  about  people.  Our 
first  CERT  facilities  were  manually  operated.  Both  the  environments  and  the 
missile  functional  tests  required  a  fairly  high  level  of  technician  training 
and  attention.  Inevitably  there  were  errors  and  lapses.  So,  we  began  to 
automate  the  tests  so  that  they  required  only  monitoring  and  a  little  button 
pushing  every  few  hours.  This  has  allowed  us  to  use  our  skilled  technicians 
on  more  productive  dvelopment  work;  while  operation  of  the  CERTs  Is  handled  by 
less  skilled  operators.  But,  It  has  not  been  an  Improvement  In  terms  of  test 
control  and  consistency.  With  so  little  required  activity,  the  operators  get 
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bored  and  fall  to  do  the  required  monitoring.  This  has  led  to  some 
interesting  data  on  missile  durability  under  extreme  environments.  For 
example,  missiles  Mill  operate  after  being  cooled  to  -90°C,  but  not  after 
being  cooled  to  -95°C.  To  avoid  obtaining  more  such  information  like  this,  we 
are  now  making  each  CERT  chamber  fully  automated. 

The  lessons  specific  to  CERT  primarily  came  In  trying  for  the  greater 
realism  which  a  CERT  should  attain.  The  fundamental  lesson  here  Is  that  there 
must  be  a  clear,  objective  audit  trail  from  measured  environments  and  service 
mission  descriptions  to  the  CERT  environments.  THe  whole  usefulness  of  CERT, 
as  opposed  to  standardized  reliability  tests,  lies  In  Its  faithfulness  to  ser- 
vice  environments.  It  Is  this  faithfulness  that  justifies  confidence  In  CERT 
results  even  before  service  results  are  available.  If  this  faithfulness  Is 
sacrificed  to  some  all-purpose  standard  or  some  waving  of  arms  and  feeling  of 
the  gut,  the  resulting  test  will  be  worthless.  It  will  be  worthless  even  If 
it  gives  the  right  results  because  no  one  will  have  confidence  that  they  are 
right.  Naturally,  there  will  be  compromises  In  the  faithfulness  of  the  test, 
compromises  necessitated  by  facility  capabilities,  funding,  schedule,  and  all 
the  other  constraints  the  test  engineer  Is  heir  to.  The  important  thing  Is 
that  these  compromises  (a)  be  defensible  as  engineering  judgements,  and  (b)  be 
documented  as  part  of  the  audit  trail. 

The  compromises  that  arise  because  of  facility  limitations  lead  to  one  of 
the  outstanding  problems  In  CERT:  When  two  environments,  desired  and 
achieved,  are  complex  and  multidimensional  as  they  are  In  CERT,  how  do  you 
objectively  quantify  their  degree  of  similarity?  It  is  anticipated  that  this 
problem  will  become  more  and  more  pressing  as  more  manufacturers  build  CERT 
facilities  because  the  question  will  arise  as  to  which  facilities  are  most 
closely  reproducing  the  desired  environment.  This  will  be  followed  by  the 
even  more  troublesome  question:  How  close  Is  close  enough?  In  the  area  of 
vibration  similitude,  we  have  developed  some  measures  at  PACMISTESTCEN  and 
eventually  we  may  be  able  to  say  how  close  Is  close  enough,  but  much  remains 
to  be  done. 

At  the  other  end  of  audit  trail  from  the  test  similitude  problem,  there 
Is  another  problem,  the  fuzzy  requirement.  Because  CERT  environments  are 
based  on  mission  profiles,  the  mission  profiles  need  to  be  defined.  Too  often 
the  Decision  Coordinating  Paper  or  Specific  Operational  Requirement  does  not  do 


this.  Then,  during  the  design  process,  various  environments  are  specified 
without  reference  to  any  missions.  Finally,  during  development,  when  a  CERT 
Is  wanted,  the  mlssslons  are  defined  for  the  first  time.  It  should  surprise  no 
one  that  the  environments  derived  from  those  missions  are  often  found  to 
conflict  with  those  specified  for  design.  The  lesson  here  Is  to  define 
missions  In  the  very  beginning. 
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SUMMARY 


From  a  management  standpoint,  the  most  Important  thing  to  realize 
about  CERT  Is  that  It  Is  only  one  part  of  a  process,  or  more  correctly,  of 
two  different  processes.  First,  It  Is  the  last  step  In  a  process  of 
environmental  definition;  a  process  which  should  begin  with  mission  defini¬ 
tion  In  the  Design  Concept  stage.  Second,  it  Is  the  discovery  step  in  the 
process  of  discovering  reliability  problems  and  correcting  them.  Without 
these  processes,  CERT  Is  an  empty  exercise.  With  them.  It  Is  a  useful  and 
important  tool. 
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INTRODUCTION 

By  now  most  of  the  avionics  community  has  heard,  at  least  once,  about 
the  success  of  the  Combined  Environment  Reliability  Test  concept  in  replicat¬ 
ing  field  failure  rates  and  modes.  This  paper  will  not  present  more  on  the 
primary  reason  that  a  Combined  Environment  Reliability  Test  should  be  a  part  of 
any  avionics  acquisition  program.  It  will,  instead,  present  for  your  consider¬ 
ation  some  other  findings  of  the  CERT  evaluation  program  and  concurrent  relia¬ 
bility  demonstration  flyoffs  that  are  significant  and  could  contribute  to  the 
reliability  and  maintainability  of  avionics  systems.  These  findings  are  im¬ 
portant  to  you  because  they  show  that  insufficient  attention  has  been  given  tc 
some  important  aspects  of  these  systems. 

The  thrust  of  this  paper  can  be  summed  up  by  paraphrasing  the  late  George 
Meany.  When  Mr.  Meany  was  asked  the  function  of  a  labor  union,  he  replied 
"to  demand  more  and  more  here  and  now."  This  paper  will  ask  that  we  attempt 
to  achieve  more  and  more  sooner  and  will  show  the  benefits  of  so  doing. 

If  you  conceive  of  any  weapons  system  and  its  subsystems  as  a  group  of 
stools,  each  doing  its  share  to  support  the  mission,  the  neglected  aspects 
mentioned  previously  will  be  apparent. 

The  legs  of  any  of  these  subsystem  "stools"  would  be  those  of  the  stool 
pictured  in  Figure  1.  Obviously  the  stability  of  the  stool  and,  therefore, 
its  support  of  the  mission  depend  on  the  quality  of  the  legs,  that  is,  the 
quality  of  the: 

System  Hardware 

Maintenance  Support  Hardware 

Software  -  Both  System  and  Maintenance  Support 

Personnel  Performance. 
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IAINT  SUPPORT  HARDWARI 


The  systems  tested  in  the  evaluation  program  and  flyoffs  had  defective 
hardware  and/or  software.  Undoubtedly,  these  deficiencies  had  not  only  thei 
direct  effects  on  maintenance  costs  but  an  indirect  one  of  lowered  personnel 
performance  as  well. 

Specifically,  it  was  found  that: 

1)  In  at  least  one  instance,  overly  difficult  system  oper¬ 
ating  procedures  were  identified  by  CERT. 

2)  Repair  and  maintenance  Technical  Orders  still  contained 
important  technical  errors  and  omissions  several  years 
after  initial  deployment  of  the  system. 

3)  Minimum  Performance  Test  (MPT)  limits  and  procedures 
that  were  as  much  as  four  years  old  still  contained 
errors,  omissions,  and  hard  to  understand  sections. 

(These  findings  show  that  CERT  can  detect  system  and  mainte¬ 
nance  support  software  flaws.) 

4)  Non-failure  causes  of  maintenance  expense  that  had  al¬ 
ready  occurred  in  the  field  were  encountered  during 
CERTs. 

5)  CNDs  occurred  during  a  CERT  at  a  rate  similar  to  that 
encountered  in  the  field. 

(Findings  4  and  5  are  two  more  examples  of  CERT's  ability  to 

detect  system  hardware  malfunctions.) 


6)  Some  test  equipment  specified  for  system  maintenance 
was  either  not  available,  less  suitable,  and/^.  less 
easy  to  use  than  other  equipments.  More  important  is 
the  fact  that  some  test  sets  had  significant  design 
and  MTBF  problems. 

(Finding  6  shows  that  CERT  detects  maintenance  support  hard¬ 
ware  problems.) 

Since  CERTs  identified  these  problems  on  already  deployed  systems,  it 
seems  reasonable  that  a  CERT  would  reveal  the  existence  of  similar  types  of 
problems  early  in  an  acquisition  program. 

The  hidden  benefits  of  a  CERT,  then,  are  that  it  will  reveal,  even  spot¬ 
light: 

1 )  System  software  problems 

2)  Defects  in  the  maintenance  support  software  and 
equipment 

3)  The  existence  of  volatile  system  hardware  problems 
(CNDs  or  RETOKs) 

as  well  as  its  primary  objective: 

Nonvolatile  system  hardware  defects. 

CERT  was  conceived  for  use  in  an  acquisition  program  as  a  reliability 
demonstration  or  growth  test  to  be  conducted  singularly  or  competitively  prior 
to  the  buy  decision.  When  CERT  or  a  CERT  is  referred  to  in  this  paper,  it  will 
be  in  this  same  context;  that  Is,  a  mission  profile  combination  of  4  to  6 


environmental  parameters  used  to  improve  or  demonstrate  reliability  in  a  long- 
duration  test. 

It  will  also  be  recommended  that  the  CERT  concept  be  applied  to  first 
article  qualification  and  production  acceptance  testing  of  parts,  modules, 
and  systems.  By  CERT  concept  is  meant  the  combination  of  environmental 
parameters,  either  mission  profile  or  reasonable  extremes,  in  a  short-to 
medium- duration  test.  It  will  be  emphasized  that  in  the  area  of  acceptance 
testing  the  CERT  concept  is  particularly  valuable  as  a  product  improvement 
check  tool.  Before  expanding  on  these  points  and  citing  occurrences  from 
the  evaluation  program  and  flyoffs,  some  background  on  the  program  itself 
would  seem  to  be  in  order. 
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BACKGROUND 


The  CERT  Evaluation  Program  was  conceived  by  members  of  the  Air  Force 
Flight  Dynamics  Laboratory  (now  Air  Force  Wright  Aeronautical  Laboratories  - 
Flight  Dynamics  Laboratory)  and  the  Aeronautical  Systems  Division  Engineer¬ 
ing  Deputate  in  1976.  'The  technical  merit  and  relative  cost  effectiveness  of 
various  combinations  of  environmental  parameters  were  to  be  determined  by  com¬ 
paring  the  failure  modes  and  rates  achieved  in  the  laboratory  to  those  exper¬ 
ienced  in  the  field  for  each  of  the  systems  tested.  These  systems  were  current 
USAF  inventory  avionics  items  tested  according  to  mission  profiles  of  various 
examples  of  the  several  categories  of  LISAF  aircraft.  Figure  2  shows  the  types 
of  equipment  tested  and  the  aircraft  from  which  the  mission  profiles  were  de¬ 
rived. 

Figure  3  shows  the  4  major  components  of  a  combined  environment  test  plan 
and  the  information  sources  for  these  components.  Each  test  was  planned  by 
an  engineer  and  an  experienced  avionics  technician  whose  prime  concern  was 
that  the  operational,  maintenance,  and  environmental  conditions  were  realistic. 

The  process  of  deriving  realistic  environmental  profiles  will  not  be 
discussed  here.  Realistic  operational  conditions  were  assured  by  use  of  the 
applicable  aircraft  flight  manuals  and  information  gathered  from  the  operational 
commands,  especially  interviews  with  pilots  and  other  operations  personnel. 

It  was  the  goal  of  the  program  to  both  operate  and  repair  the  avionics 
systems  using  normal  flight  line  and  field  shop  test  sets.  The  environment  to 
which  both  the  operating  and  repair  test  sets  were  exposed  was  that  of  a  normal 
field  shop.  But  the  useage  rate  of  the  operating  sets  was  considerably  accel¬ 
erated  compared  to  normal.  Due  to  the  compression  of  ground  park  time,  these 
sets  were  operated  24  hours  per  day,  5  or  6  days  each  week,  for  approximately 
two  months  per  test  sequence. 
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CERT  EVALUATION  PROGRAM 


EQUIPMENT  TYPES  TESTED 

COMMUNICATIONS  RADIOS 
(2) 

IFF’s 

(2) 

IMU  (1) 

NAVIGATION  RADAR  (1) 

NAVIGATION  RADIOS 
(2) 


MISSION  PROFILES  FROM 

B52 

F5 

FBI 1 1 A 
AIOA 

FI  5  (BOTH  EQUIPMENTS) 
A7D 
B52 

Cl  30 
F5 

FBI 11 A 


Figure  2 
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Figure 
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Each  test  plan  required  that  steps  in  the  trouble  isolation  process  to  an 
LRU  at  the  chamber,  or  to  an  SRU  at  the  CERT  repair  shop,  be  completely  in 
accordance  with  the  planners'  understanding  of  flight  line  and  field  shop 
procedures.  This  understanding  was  gained  from  applicable  Technical  Orders 
and  information  gathered  from  the  operating  commands,  especially  Interviews  with 
maintenance  officers,  and  flight  line  and  field  shop  personnel. 

Only  in  the  area  of  SRU  logistics  was  there  significant  deviation  from 
normal  procedures.  In  order  to  determine  the  Called  part(s)  and  have  the  fail¬ 
ure  mode  confirmed,  failed  SRUs  were  shipped  directly  to  the  depot,  repaired, 
and  returned  directly  to  the  CERT  facility.  The  depot  also  returned  the  failed 
parts  along  with  a  report  on  their  troubleshooting  and  repair  for  use  In  anal¬ 
ysis. 

While  the  CERT  Evaluation  Program  was  In  progress,  two  reliability  demon¬ 
stration  flyoffs  were  conducted  In  the  CERT  facilities.  The  flyoffs  consisted 
of  operating  the  competing  manufacturers '  systems  under  environmental  and  oper¬ 
ational  profiles  derived  for  the  aircraft  to  which  the  system  was  to  be  fitted. 
The  repair  procedures  during  these  flyoffs  differed  from  those  in  the  evaluation 
program  in  that  the  manufacturers'  technicians  performed  all  maintenance  and 
design  changes  after  a  failure  was  permitted.  The  Information  from  these 
CERTs  was  used  to  establish  relative  life-cycle  costs  for  the  various  manufac¬ 
turers'  systems  and  Identify  areas  for  product  Improvement. 

From  the  above,  it  can  be  seen  that  a  CERT  gives  a  trained  team  the  oppor¬ 
tunity  to  understand,  operate,  and  maintain  an  avionics  system  in  a  human  en¬ 
vironment  that  is  relatively  free  of  distractions  and  pressures,  yet  realistic 
as  to  procedures,  methods,  and  equipment  operational  environment. 

With  this  look  at  the  evaluation  program  as  background,  let  us  now  con¬ 
sider  the  findings  of  this  program  in  more  detail. 
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EXPANSION  AND  EXAMPLES 


1 .  Improved  Operating  Procedures 

While  simplicity  of  operation  is  one  of  the  prime  goals  of  systems  de¬ 
signers,  this  goal  is  often  missed,  partly  due  to  their  level  of  expertise  and 
familiarity  with  the  system.  A  CERT  gives  an  opportunity  for  evaluation  under 
realistic  operational  conditions  with  trained  operators  and  evaluators  who 
have  had  to  learn  the  system.  This  learning  is  guided  by  the  same  flight  man¬ 
uals,  Technical  Orders,  and  other  system  software  that  guide  the  learning  ex¬ 
perience  of  field  personnel.  The  system  operation  during  the  CERT  is  controlled 
by  the  same  system  software  and  firmware  used  in  field  units.  An  example  of 
CERT's  effect  can  be  found  in  one  of  the  CERT  flyoffs  of  competing  systems. 

One  of  the  systems  tested  had  a  long  and  complex  initialization  procedure.  It 
is  our  understanding  that  the  version  of  this  system  finally  deployed  was  much 
easier  to  initialize. 


2.  Technical  Order  Defects 

During  test  planning,  the  approved  Technical  Orders  are  the  major  source 
of  the  following  information: 

Theory  of  Operation 

Wiring  Diagrams  for  Unit  Hook-up 

Recommended  Troubleshooting  and  Maintenance  Methods. 

Approved  flight  manuals  are  used  to  determine  when  and  how  the  system  was  used. 
It  was  often  found  that  the  Technical  Orders  did  not  contain  sufficient  Infor¬ 
mation  to  allow  hook-up  and  operation  of  the  system.  This  missing  information 
would  have  assisted  field  personnel  In  troubleshooting  and  retrofitting.  For 
example,  a  communications  radio  was  found  to  employ  positive  and  negative  logic 
in  various  sections.  The  T.O.  would  speak  of  logic  "0"  or  logic  M1H  without 
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giving  any  clue  as  to  the  actual  voltage  condition.  Without  this  signal 
level  information  on  lines  external  to  the  LRUs,  a  test  setup  was  not  possi¬ 
ble.  It  is  felt  that  the  field  is  forced  to  look  elsewhere  for  the  information, 
when  it  is  needed,  with  an  attendant  loss  of  time  and  increased  cost. 

3.  Minimum  Performance  Test  Defects 

During  test  planning  the  Minimum  Performance  Test  procedures  are  reviewed 
to  see  if  they  do,  indeed,  fully  check  out  the  system's  minimum  performance. 

The  MPT  limits,  built-in  test  (BIT)  limits,  as  well  as  knowledge  of  the  opera¬ 
tional  requirements  are  used  to  define  a  failure.  Once  the  test  articles  are 
received  the  repair  technician  performs  an  MPT  on  each  unit  and  during  the 
progress  of  the  test  MPTs  are  performed  on  each  failed  unit.  This,  in  effect, 
is  a  proof  test  of  the  T.O.  and  problem  areas  are  quickly  identified.  For 
example,  during  preparation  for  a  test  of  an  IFF  transponder  the  repair  technician 
determined  that  some  parts  of  the  procedure  were  unclear,  others  needlessly 
difficult,  and  some  did  not  accomplish  the  desired  result.  With  the  concurrence 
of  the  test  engineer,  he  rewrote  the  procedure  for  his  own  use.  During  the 
course  of  the  test,  T.O.  changes  corrected  50%  of  all  defects  and  90%  of  the 
very  serious  ones.  It  took  a  little  over  five  months  from  the  time  test  planning 
began  to  completion  of  the  revision;  at  this  point  the  T.O.s  were  four  years 
old.  A  CERT  during  this  acquisition  could  have  resulted  in  early  identifica¬ 
tion  of  these  deficiencies  and  four  years  of  reduced  maintenance  expense. 

In  this  same  test,  it  was  found  that  several  failures  could  have  been 
cleared  by  an  adjustment  rather  than  SRU  replacement;  this  adjustment  was 
authorized  at  intermediate  level  but  not  adequately  referenced  in  the  T.O.  . 

How  many  field  shops  also  failed  to  make  this  connection  and  how  much  this  cost 
the  Air  Force  cannot  easily  be  determined,  but  a  CERT  during  the  acquisition 


program  could  have  avoided  these  costs.  These  three  examples  show  that  CERT 
can  Identify  system  and  maintenance  support  software  problems.  Now  let  us 
take  a  look  at  some  hardware  problem  examples. 

4.  Identification  of  "Nonfailure11  Causes  of  Maintenance  Expense 

Data  from  several  CERTs  Indicates  that  environmentally-caused  cost  drivers 
that  are  not  usually  discovered  as  system  failures  or  are  not  counted  as  system 
failures  under  AFM  66-1  were  also  identified  by  CERT.  Some  examples  are: 
vibration-shortened  light  bulb  life,  shock  mount  breakage,  P.C.  board  delami¬ 
nation,  corrosion  of  metal  parts  and  deposition  of  corrosion  on  other  parts. 
Field  shop  Chiefs  were  queried  concerning  board  delamination  and  corrosion 
deposition  problems  on  the  LRUs  that  had  exhibited  these  problems  during 
CERT;  they  confirmed  their  existence  in  the  field. 

5.  Occurrence  of  RETQKs 

During  the  course  of  the  evaluation  program,  it  was  noted  that  the  CERT 
flight  line  CND  and  field  shop  RETOK  occurrence  rates  seemed  to  parallel  field 
rates.  An  attempt  to  correlate  field  and  CERT  CND  modes  was  inconclusive  due 
to  the  flight  line  test  methods  used  in  the  field  and  at  the  CERT  facility  on 
the  systems  available  for  comparison.  A  recent  study  was  made  of  CERT  as  a 
RETOK  reduction  tool.  During  this  study  seven  of  the  eleven  units  under  test 
exhibited  malfunctions  that  were  not  apparent  in  the  normal  shop  environment. 
Five  of  the  seven  exhibited  the  malfunction  exhibited  in  the  aircraft.  All 
malfunctions  appeared  within  24  hours  of  test,  the  mean  time  being  5.9  hours. 

Two  CND  problems  that  occurred  during  one  of  the  evaluation  program  tests 
were  tracked  to  their  root  causes.  They  are  presented  to  give  the  reader  an 
idea  of  the  environmentally-caused  problems  that  are  being  experienced  with 
avionics  equipment. 
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1.  An  IFF  system  repeatedly  was  out  of  W>T  and  BIT  self¬ 
test  frequency  tolerances  after  a  1/2-hour  cold  soak  at 
about  -65° F.  It  was  proved  that  the  L-band  source  had  a 
positive  temperature  coefficient  of  frequency  that 
caused  the  BIT  tolerances  to  be  exceeded  at  case  temper¬ 
atures  below  -20°F.  Assuming  that  BIT  self-test  test 
limits  reflect  the  band  width  of  the  interrogator- 
responsor,  loss  of  range  must  occur  in  Arctic  and  high- 
altitude  environments. 

2.  Mission  profiles  for  the  APX-101/A-10A  test  called  for 
85%  relative  humidity  at  a  temperature  of  90°F  during 
the  hot  soak.  When  the  facility  was  operating  at  the 
limit  of  allowed  tolerances,  it  sometimes  produced  an  RH 
of  90%.  When  this  condition  was  encountered  after  a 
cold  flight  (0  to  -16°F),  as  It  might  be  by  an  A-10 
descending  from  cross-country  altitude  near  Myrtle  Beach 
or  other  tropical-subtropical  base,  the  dlplexer  and 
diversity  switch  were  coated  inside  and  out  with  water 
and  a  loss  of  output  resulted.  Water  Inside  the  equip¬ 
ment  bays  and  IFFs  on  landing  was  confirmed  by  Myrtle 
Beach  personnel. 

These  four  examples  -  the  broken  shock  mount,  the  frequency  shift,  the 
condensation  failure,  and  the  RETOK  reduction  study  -  show  how  a  CERT  can 
identify  "nonfailure,"  CND,  and  RETOK  problems  in  system  hardware. 
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6.  Test  Equipment  Checkout 

Perhaps  the  biggest  advantage  of  a  CERT  Is  that  you  can  have  two  different 
environmental  reliability  demonstrations  for  the  price  of  one.  All  CERTs  would 
use  the  authorized  flight  line  and  field  shop  test  equipment  and  sets  for  check 
out  and  repair  of  the  test  articles.  Host,  if  not  all,  would  use  these  same 
items  for  operation  as  well.  In  all  of  the  CERT  evaluation  program  tests, 
failure  data  on  the  test  equipment/ sets  was  accumulated.  This  data  pinpointed 
several  Items  that  exhibited  poor  reliability  when  compared  to  the  average  of 
all  items  used.  One  example  Is  the  navigation  test  set  that  exhibited  such 
severe  attenuator  hand-effects  that  the  set  was  almost  unusable  for  sensi¬ 
tivity  checks.  This  problem  was  temporarily  eliminated  by  PMEL  on  some  of  the 
sets.  Another  example  Is  the  counter-converter  that  produced  false  counts 
under  simultaneous  conditions  of  near  maximum  ambient  temperature  (55°F)  and 
near  maximum  input  level.  Failure  information  on  test  sets  is  not  generally 
available,  so  confirmation  of  the  hand-effect  problem  was  sought  from  field 
personnel  and  obtained. 

The  test  planning  stage  of  CERTs  has  also  provided  information  about  the 
test  equipment  specified  for  field  use.  In  planning  for  a  test  of  a  communica¬ 
tions  radio,  the  test  engineer  found: 

a)  That  R.F.  pads  specified  were  not  available,  had  not  been  available 
a  year  before  when  the  T.O.  was  published,  and  probably  were  not 
available  when  the  T.O.  was  written. 

b)  That  laboratory-grade  equipment  was  specified  when  service-grade 
equipment,  proven  by  commercial  usage  on  equivalent  land  mobile 
service  equipment,  was  available  at  considerably  less  cost.  One 
example  Is  the  use  of  laboratory-quality  signal  generators  and 
modulation  monitors  Instead  of  commercial  communications  monitors. 
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c)  That  the  service-grade  equipment,  designed  for  more  specific  purposes. 

Is  easier  to  operate  than  the  more  flexible  laboratory-grade  equipment. 

An  excellent  example  of  this  finding  as  well  as  of  b)  above  would  be 
using  a  commercial  signal-to-noise  ratio  measurement  set  sold  as  the 
"Sineadder"  rather  than  a  laboratory  distortion  analyzer. 

These  examples  show  how  a  CERT  gives  an  opportunity  for  thorough  evaluation 
of  the  test  sets  and  equipment  used  for  flight  line  or  field  maintenance. 

In  a  recent  address  to  the  IEEE  at  NAECON,  ASD  Commander,  General  Skantze, 
pointed  out  the  need  for  moving  reliability,  maintainability,  and  product  assurance 
considerations  to  an  earlier  time  in  the  acquisition  process.  The  hidden  benefits 
of  a  CERT  are  that  it  gives  the  opportunity  to  examine  many  aspects  of  the 
reliability /maintainability  question  that  are  often  neglected  before  a  buy 
decision  is  made.  A  CERT,  whether  used  as  a  reliability  growth  or  demonstration 
test,  can  provide  accurate  data  to  support: 

Life  Cycle  Costing. 

System  Hardware  Improvement. 

Maintenance  Support  Hardware  Evaluation  and  Improvement. 

System  and  Maintenance  Support  Software  Evaluation  and  Improvement. 
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OTHER  USES  OF  THE  CERT  CONCEPT 

Based  on  evaluation  program  results,  the  CERT  concept  recommends  Itself 
for  use  in: 

Qualification  Testing 
Acceptance  Testing 

of  parts,  modules,  and  systems.  A  very  important  special  case  of  acceptance 
testing  is  the  acceptance  of  a  product  improvement.  Consider  the  case  of  the 
series  of  micro-miniature  connectors  that  was  improved  by  replacing  the  easily 
broken  plastic  mounting  ring  with  one  made  of  metal.  Another  change  had  to 
be  made  when  it  was  discovered  that  the  metal  ring  caused  excessive  conden¬ 
sation  and  corrosion  inside  the  connector.  A  short-duration  Combined  Environ¬ 
ment  Test  (CET)  could  have  determined  this  before,  rather  than  after,  acceptance 
and  deployment.  In  most  qualification  and  acceptance  testing,  combinations  of 
realistic  environmental  parameter  extremes,  rather  than  mission  profile  com¬ 
binations,  should  provide  the  most  useable  results. 

Combined  Environments  In  acceptance  and  qualification  testing  offers  many 
of  the  same  advantages  realized  in  a  CERT.  The  following  are  especially  worthy 
of  note: 

You  save  time  -  as  compared  to  sequential  tests  on  the 
same  articles. 

You  save  money  -  on  test  article  and  facility  operating 
cost  as  compared  to  application  of 
single  environments  to  different  test 
articles. 

You  gain  synergy  -  of  whatever  type  and  In  whatever  amount 
exists  during  your  profiles. 
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Why  take  the  time  to  run  individually  the  several  environmental  stress 
tests  such  as  temperature  cycling,  humidity  exposure,  vibration  exposure, 
altitude  cycling,  and  electrical  discharge  exposure  that  could  be  combined? 

If  the  article  passes  the  CET,  you  know  that  the  article  will  survive  not  only 
the  individual  environmental  parameters  but  their  synergistic  force  as  well. 
And  if  the  article  fails,  the  cause,  if  needed,  will  be  apparent  in  most, 
if  not  all,  carefully  designed  tests. 


SUMMARY 


The  hidden  benefits  of  a  CERT  are  that  it  reveals: 

System  Software  Problems 

Defects  in  Maintenance  Support  Hardware  and  Software 

High  CND  Rate  Systems. 

It  does  this  while  giving  you  an  accurate  value  for  system  hardware  failure 
rates  and  modes. 

In  closing,  the  author  would  like  to  point  out  his  agreement  with  those 
who  say  that  reliability  and  maintainability  are  obtained  only  by  applying 
your  "best  engineering  practices"  during  design. 

Then . 

You  test  . 

To  be  certain  . 

That  your  design  judgments  were  accurate  and  economical,  that  quality 
is  maintained  in  production,  and  to  develop  the  next  generation  of  "best  eng 
neering  practices." 

The  evidence  shows  that  the  CERT  concept  provides  the  most  accurate, 
most  comprehensive,  most  economical  means,  presently  available,  to  do  this 
testing. 
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INTRODUCTION 


The  level  of  reliability  achieved  by  deployed  avionics  equipment  has 
generally  been  less  than  desired  by  the  acquisition,  logistics  and  using 
commands.  Studies  have  shown  that  there  are  numerous  and  often  overlapping 
factors  that  contribute  to  this  problem  (Refs.  1,2  and  3).  Additionally,  these 
factors  are  not  Independent  of  one  another  so  that  It  is  difficult  to  easily 
identify  how  much  avionics  equipment  unreliability  Is  due  to  any  one  specific 
factor. 

One  of  the  studies  grouped  these  numerous  factors  Into  the  following 
six  broad  basic  areas  (Ref.  3): 

1.  Reliability  testing  and  analysis 

2.  Field  data  collection 

3.  Reliability  management  and  documentation 

4.  Reliability  design 

5.  Reliability  training 

6.  Software  reliability. 

It  Is  the  thrust  of  this  paper  to  show  how  the  use  of  a  properly  structured 
testing  program  will  cause  significant  equipment  reliability  Improvements  In 
many  of  these  basic  areas.  The  paper  will  discuss  these  Improvements  In  terms 
that  are  meaningful  and  measurable  to  the  decision  makers  that  exist  In  the 
life  cycle  of  a  typical  avionics  equipment  system.  For  the  sake  of  discussion, 
these  decision  makers  can  be  represented  as  the  acqulsltor,  logistician,  opera¬ 
tional  user,  and  command  level  of  the  avionics  equipment  system.  To  understand 
the  perspective  of  these  decision  makers,  the  next  section  outlines  what  is 
meaningful  and  generally  used  to  measure  their  performance.  The  primary  drivers 
for  these  decision  makers  are: 

Acqulsltor:  The  primary  drivers  In  an  equipment  acquisition  program 
have  been  performance,  cost  and  schedule  (Ref.  4).  Reliability  may  become  a 
major  consideration  because  of  the  recent  issuance  of  DoO  Directive  5000.40. 
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Logistician:  The  primary  drivers  are  providing,  vtoen  and  vtoere  needed, 
adequate  quantities  of  support  equipment,  spare  parts,  trained  personnel 
and  equipment. 

Operational  User:  The  primary  drivers  are  to  have  avionics  equipment 
that  have  high  levels  of  availability  and  sustainability  that  allow  the 
accomplishment  of  the  required  missions. 

Command  Level:  The  primary  drivers  are  to  produce  more  operational 
capability  at  a  minimum  overall  life-cycle  cost. 

ACQU1SIT0R 


The  acqulsltor  decision  maker  generally  wants  to  reduce  the  risks  of 
schedule,  resources,  or  technical  problems.  This  means  that  any  deficien¬ 
cies  that  slip  through  the  design  process  need  to  be  Identified  as  early 
as  possible  since  It  Is  easier  and  cheaper  to  change  paper  designs  than 
rework  already  built  hardware.  There  are  two  ways  of  reducing  the  probabi¬ 
lity  that  a  design  defect  will  go  into  hardware:  (1)  strengthen  the  design 
techniques  and  thereby  reduce  the  number  of  design  defects,  and  (2) 
strengthen  the  evaluation  techniques. 

Electronic  hardware  technology  Is  constantly  changing  and  the  develop¬ 
ment  of  necessary  reliability  data  base  on  new  technology  components  Is  not 
keeping  pace.  Additionally,  the  Increased  complexity  of  modem  avionics 
Increases  the  chance  of  undetected  design  deficiencies  occurring  in  a 
design.  Both  of  these  factors  tend  to  suggest  that  to  reduce  program  risk, 
techniques  In  addition  to  design  are  needed.  This  conclusion  Is  supported 
by  NASA  Apollo  spacecraft  experience:  "The  single  most  important  factor 
leading  to  the  high  degree  of  reliability  of  the  Apollo  spacecraft  was  the 
tremendous  depth  and  breadth  of  the  test  activity."  (Ref.  5). 


( 
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The  thrust  of  such  testing  should  be  to  demonstrate  that  the  design  Is 
proper  and  will  perform  properly  In  all  deployment  environments,  thereby 
reducing  risks  of  schedule  and  resources  overruns. 

LOGISTICIAN 


The  logistician  has  a  different  view  of  reliability  than  the 
acqulsltor.  Resources  are  the  driving  factor  behind  the  logistician.  It  Is 
his  job  to  maintain  adequate  spares  to  the  operational  units  and  the 
necessary  support  equipment  needed  for  maintenance.  He  sees  the  reliabi¬ 
lity  of  a  new  system  as  how  many  spares  must  be  obtained. 

His  criteria  for  failure  classification  are  primarily  resource  and 
operation  impact  oriented.  This  Is  In  contrast  to  the  acquisition  com¬ 
munity  which  Is  primarily  mission  success  oriented.  To  the  acquisition 
world,  the  reliability  of  an  avionics  system  Is  judged  as  If  the  system  was 
Its  own  separate  entity.  Thus,  any  evidence  of  externally  Induced 
failures  or  secondary  failures  generally  would  be  classified  as  nonrele- 
vant.  Yet,  these  same  failures  would  be  classified  as  relevant  by  the 
logistics  or  operational  comnunity  because  the  failure  consumed  resources 
and  reduced  operational  capability.  With  his  budget  constraints,  the 
logistician  needs  to  be  able  to  know  the  number  of  spares  and  support 
equipment  he  must  Initially  obtain  In  order  to  support  the  operational 
units.  He  needs  to  know  from  a  reliability  test  a  fairly  accurate  MTBF 
(mean  time  between  failure)  and  what  will  be  falling  In  order  to  allocate 
his  limited  funds  toward  spares  and  support  costs.  Therefore,  the  logisti¬ 
cian  views  the  success  of  a  reliability  test  concept  by  Its  ability  to  pro¬ 
vide  data  to  help  him  accurately  predict  his  sparing  requirements. 


OPERATIONAL  USER 


The  operational  user  considers  reliability  as  to  how  It  Impacts  main¬ 
tainability  and  availability  of  the  total  weapons  system.  A  maintenance 
squadron's  scorecard  is  determined  by  how  many  aircraft  are  operationally 
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ready  at  any  given  time.  Reliability  Information  to  this  community  Is  used 
to  determine  a  system's  availability  as  It  Impacts  downtime  per  sortie  and 
the  resources  needed  to  handle  maintenance  events.  It  takes  time  and  man¬ 
power  to  troubleshoot, diagnose  and  repair  failures;  and  the  more  compli¬ 
cated  the  equipment,  the  longer  It  takes  to  repair.  The  time  spent  on 
avionics  equipment  maintenance  Is  time  the  aircraft  Is  not  available  to  per¬ 
form  Its  mission.  Therefore,  the  operational  user  determines  the  reliabi¬ 
lity  of  a  system  by  the  amount  of  aircraft  downtime  It  causes.  The 
allocation  of  resources,  tools,  equipment,  skills  and  manpower  at  the  base 
level  ;is  performed  In  order  to  produce  a  maintenance  system  tailored  to 
product  more  combat  sorties  with  the  available  resources.  The  impact  of 
reliability  testing  on  aircraft  downtime  Is  the  criterion  the  operating 
commands  would  use  to  judge  its  success  rate. 

COMMAND  LEVEL 

The  decision  maker  at  the  command  level  sees  the  big  picture.  He  wants 
to  provide  the  operational  users  what  they  need  at  minimum  overall  (life 
cycle)  cost.  Therefore,  the  command  level  decision  maker  tries  to  balance 
the  resources  spent  by  the  acqulsltor,  logistician  and  operational  user  to 
minimize  the  total  cost  of  the  avionics  equipment  system.  The  ability  of 
Improved  testing  to  help  reduce  overall  avionics  equipment  costs  would  be 
how  the  decision  maker  at  command  level  would  view  Improved  testing. 


BASIS  FOR  MISSION  PROFILE 
COMBINED  ENVIRONMENT  RELIABILITY  TEST  (CERT) 

Analysis  of  environmental  data  show  that  the  environmental  conditions 
of  temperature,  humidity,  altitude,  vibration  and  cooling  airflow  do  not 
generally  remain  constant  throughout  an  aircraft's  mission.  In  fact, 
strong  correlation  exists  between  the  mission  of  an  aircraft  and  the 
environmental  stress  combinations  that  are  Imposed  upon  Its  avionics  at  aqy 
given  Instant  of  time  throughout  Its  flight.  This  correlation  between  the 
aircraft  flight  conditions  and  the  environment  In  vhlch  Its  avionics  must 


function  suggests  that  the  environmental  combinations  should  be  time 
sequenced  In  a  test  similar  to  how  the  aircraft  Is  flown  so  that  the 
resultant  laboratory  test  conditions  would  be  more  representative  of  the 
field  environment  (Refs  2,6,7).  This  concept  has  been  called  combined 
environment  profile  testing.  This  concept  forms  the  basis  for  the  CERT 
approach. 

Combinations  of  environmental  conditions  characteristic  of  those  that 
occur  during  aircraft  ground  park,  takeoff,  cruise,  combat,  and  other  major 
flight  conditions  are  put  together  Into  a  test  sequence  that  follows  the 
order  In  which  they  would  occur  during  a  typical  aircraft  mission. 
Therefore,  the  equipment  acts  as  If  It  Is  actually  undergoing  a  flight 
during  the  combined  environment  mission  profile  test  sequence.  Thus,  the 
behavior  of  the  equipment  under  test  should  closely  approach  Its  field 
performance. 

Benefits  to  Acquisition  Decision  Maker 

Avionics  technology  Is  constantly  changing.  Generally,  there  Is  very 
little  historical  data  on  how  new  technology  electronic  components  behave 
under  the  kinds  of  dynamic  environmental  stresses  to  the  experienced  In 
operational  deployment.  Therefore,  evaluation  of  the  anticipated  designs 
need  to  be  accomplished  early  In  the  design  process.  Additionally,  these 
tests  need  to  be  realistic  so  that  there  Is  a  high  degree  of  confidence  by 
the  decision  maker  that  the  Identified  defects.  If  not  corrected,  would 
occur  In  the  deployment  environment. 

CERT  has  such  credibility.  This  comes  about  because.  In  CERT,  the 
environmental  stresses  are  combined  and  sequenced  In  a  realistic  temporal 
manner  as  If  the  test  Item  was  In  actual  flight.  Additionally,  the 
approach  has  several  conceptual  advances  that  tend  to  help  give  credibility 
to  a  decision  maker:  (1)  the  environmental  stresses  occur  simultaneously  as 
If  in  deployment,  (2)  no  engineering  protest  judgement  needs  to  be  made  as 
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to  which  order  or  sequence  of  separate  single  environment  tests  would  be 
appropriate,  and  (3)  test  conditions  can  be  directly  related  to  operational 
usage  with  no  artificial  raising  of  test  levels  above  deployment  stress 
levels. 

A  way  of  utilizing  CERT  to  accomplish  this  objective  of  early  Iden¬ 
tification  of  deficiencies.  Is  through  a  test-anal yze-f lx  growth  test 
program  which  uses  CERT  environmental  test  conditions.  Analysis  of  CERT 
growth  programs  used  on  five  different  avionics  equipment  systems  found 
that  the  reliability  of  these  equipments  increased  at  a  rate  of  300  percent 
for  each  Increase  In  order  of  magnitude  In  number  of  test  hours  (Refs.  9  and 
10).  This  significant  Increase  In  equipment  reliability  occurred  In  less 
than  four  calendar  months.  In  fact,  the  savings  in  equipment  development 
time  of  two  to  four  years  have  been  projected  by  a  major  electronics  equip¬ 
ment  contractor  (Ref  17).  These  analyses  were  used  as  a  basis  for  company 
funding  for  Installation  of  a  CERT  facility. 

A  CERT  growth  test  In  addition  to  providing  highly  credible  results  In 
a  short  period  of  time  can  provide  significant  other  savings  through  the 
deletion  of  separate  Individual  tests.  For  example,  one  Air  Force  equip¬ 
ment  acquisition  program  eliminated  seven  separate  single  environmental 
qualification  tests  and  the  normal  reliability  demonstration  test  with  a 
single  combined  environment  mission  profile  test  (Ref.  9).  The  cost 
savings  realized  by  this  approach  can  be  estimated  using  the  HQ  Air  Force 

Systems  Command  (AFSC/XR)  figures  for  test  costs  (Ref.  11).  Table  I  shows 

the  cost  savings  based  on  these  cost  factors  and  the  cost  to  conduct  a 
CERT  growth  test  of  approximately  2,000  test  hours.  The  cost  savings  are 
shown  In  terms  of  dollars  saved  and  number  of  calendar  days.  After  the 
approximately  2,000  hours  of  CERT  growth  testing,  the  same  serial  numnber 
units  were  used  directly  In  additional  flight  testing. 

The  data  In  Table  I  are  from  a  competitive  CERT  growth  program  with 
three  separate  contractors  being  tested  at  a  single  site.  The  overall  cost 
effectiveness  of  this  approach  to  the  acquisition  decision  maker  Is  sum* 

marlzed  In  Table  II.  The  savings  of  over  $800,000  are  dollars  that  the 

acquisition  decision  maker  can  reprogram  or  turn  back  to  the  Air  Force. 
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With  this  reduced  expenditure  of  resources  In  the  acquisition  phase  of 
this  program,  there  Is  no  apparent  compromise  In  fielded  system  perfor¬ 
mance  as  demonstrated  by  Its  recorded  1192-hour  field  MTBF,  vahich  Is  2.4 
times  the  design  goal  of  500-hour  MTBF. 

The  productivity  of  this  approach  has  been  recognized  and  has  been 
used  on  another  Air  Force  program  (Ref.  10)  and  will  be  used  on  the  Global 
Positioning  Satellite  (GPS)  system  user  equipment. 

Benefits  to  Logistician 

The  MTBF  of  a  new  system  is  Important  to  a  logistician.  This  value  is 
used  as  a  major  consideration  in  the  calculatlpns  to  determine  the  number  of 
spares  to  be  procured.  It  is  the  logistician's  job  to  ensure  that  adequate 
resources  are  made  available  to  each  operational  unit.  Otherwise,  a  unit 
with  inadequate  resources  Is  placed  on  NORS  (not  operationally  ready 
spares)  status.  This  Impacts  the  operational  readiness  of  the  Air  Force 
fleet.  CERT  can  be  of  great  benefit  to  the  logistician  by  more  closely 
predicting  the  MTBF  of  a  new  system.  Previous  reliability  tests  have 
demonstrated  an  unrealistic  MTBF  which  has  led  to  inadequate  spares  pro¬ 
curement.  CERT  has  shown  a  much  closer  correlation  between  the  laboratory 
test  and  field  conditions.  By  more  closely  duplicating  the  aircraft's 
operational  environment,  CERT  gives  a  more  realistic  estimate  of  a  new 
system's  MTBF  (Ref.  12). 

Air  Force  laboratory  tests  have  demonstrated  that  mission  profile 
testing  shows  MTBF  values  within  70%  of  actual  field  data,  which  leads  to  a 
1.9  to  1.0  laboratory  to  field  ratio.  By  predicting  a  viable  MTBF  value, 
mission  profile  testing  will  enable  the  logistician  to  know  the  true  number 
of  spares  and  resources  he  needs  to  obtain  to  fully  support  operational 
units  without  undergoing  NORS  status.  This  also  will  enable  him  to  pro¬ 
perly  allocate  and  utilize  Ms  financial  resources  to  the  fullest  capability. 
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A  recent  study  found  that  a  CERT  of  1200  hours,  typically  three 
months,  on  the  average  Mill  Identify  sufficient  numbers  of  hardMare  defl 
clencles  to  make  such  a  test  cost  effective.  Within  tMO  years.  It  Mill 
return.  In  reduced  logistics  costs,  the  Investment  In  the  costs  associated  with 
conducting  the  CERT,  performing  failure  analysis,  designing  corrective  actions, 
confirming  corrective  actions,  and  Implementing  redesign  In  the  production 
run  (Ref.  13).  This  suggests  that  an  Investment  In  a  feM  months  of  CERT  on  even 
already  deployed  systems  can  significantly  cut  Its  operating  and  support  (04S) 
costs. 

Benefits  to  Operational  User 


The  operational  user  has  different  requirements  that  reliability 
testing  needs  to  address.  A  maintenance  unit  must  keep  the  operational 
squadrons  In  readiness  by  keeping  the  doMntlme  betMeen  sorties  to  a  mini¬ 
mum.  Therefore,  any  system  requiring  repair  and  corrective  actions  prohi¬ 
bits  the  operational  user  from  achieving  his  goal. 

It  takes  time  and  resources  to  perform  maintenance  actions  on  equip¬ 
ment,  and  not  all  reported  failures  are  Identifiable.  According  to  a  recent 
report  (Ref  14),  approximately  one- third  of  all  flight  failures 
cannot  be  confirmed  at  base  or  depot  repair  facilities.  These  Intermit¬ 
tent,  or  CNDs  (cannot  duplicate),  are  typically  returned  to  the  spares 
Inventory  Mhere  they  may  re-occur.  These  intermittent  failures  constitute  a 
drain  on  resources,  increase  spares  requirements,  and  decrease  mission  suc¬ 
cess  rates.  Avionics  equipment  Is  pulled  and  sent  back  to  the  depot  for 
repair  that  may  not  need  repair  or  could  have  been  corrected  at  the  base. 
This  creates  a  congestion  In  the  supply  pipeline  by  Increasing  the  spares 
requirement.  It  also  drains  the  operational  user  by  repetitious  removal  of 
equipment,  excess  maintenance,  and  frustration  at  not  properly  diagnosing 
and  correcting  the  problem.  Mission  profile  testing  can  help  correct  this 
by  shoMlng  potential  design  deficiencies  In  new  avionics  systems  which 
can  be  corrected  before  It  reaches  the  operational  field. 
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During  the  CERT  Evaluation  Program,  mission  profile  testing  exhibited 
failures  In  which  41  percent  were  Intermittent.  By  correcting  these 
problems  early  In  the  acquisition  of  a  new  system,  the  CND  rate  can  be 
decreased  significantly.  This  will  enable  the  operational  user  to  utilize  his 
resources  efficiently  In  correcting  hard  failures  quickly  and  effectively. 

Benefits  to  Command  Level 

The  decision  maker  at  the  command  level  is  Interested  In  obtaining  the 
most  effective  utilization  of  resources  spent  by  the  acqulsltor,  logisti¬ 
cian  and  operational  user.  Studies  have  shown  that  the  earlier  the 
testing  Is  done  in  the  equipment  development  phase,  the  more  leverage  there 
Is  for  reduction  of  equipment  operation  and  support  (04S)  costs  (Ref.  ?). 
Programs,  such  as  summarized  In  Table  II,  have  shown  that  a  CERT  growth 
test  can  significantly  reduce  acquisition  costs  over  traditional  approaches 
of  waiting  until  later  In  the  development  cycle.  The  leverage  of  cost 
reduction  benefits  everyone  In  the  decision  making  process  by  reducing  overall 
equipment  costs. 

SUMMARY 


The  perspective  of  the  different  decision  makers  In  the  life-cycle  of 
a  typical  avionics  equipment  are  focused  on  factors  that  are  measurable 
and  meaningful  during  their  periods  of  accountability.  To  cause  change  in 
equipment  reliability  or  quality  requires  either  changing  the  factors  on 
which  each  decision  maker  Is  evaluated  or  to  develop  tools  that  Improve  his 
performance  when  measured  against  the  established  performance  factors. 

This  paper  discusses  how  a  combined  environments  mission  profile  (1.  e.  CERT) 
test,  when  used  as  a  test-analyze-flx  growth  test  program  In  the  acquisition 
process,  benefits  all  the  decision  makers  during  the  life-cycle  of  the 
equipment. 


TABLE  I.  SAVINGS  AVAILABLE  USING  A  CERT  GROWTH  TEST 


TEST  DELETED /REPLACED 

SAVINGS 

Deleted  seven  separate  environmental  qualification  tests 

-  $195,400 

Deleted  reliability  demonstration  test  (5000  hrs  @  $1 70/hr) 

-  850,000 

Replace  with  a  single  CERT  growth  test  {Ref.  15) 

+  230,000 

COST  SAVINGS 

$815,400 

Deleted  seven  separate  environmental  qualification  tests 

-  129  days 

Deleted  reliability  demonstration  test 

-  125  days 

Replace  with  a  single  CERT  growth  test  (Ref.  9) 

+  50  days 

NUMBER  OF  DAYS  SAVED 

204  days 

TABLE  II.  TOTAL  ACQUISITION  PROGRAM  COST  COMPARISONS 


ACTIVITY 

- cnn - 

PROGRAM 

COST 

1  Had 111 UNAL - 

PROGRAM 

COST 

1.  Design  and  build  seven  preproduction 
units,  support  government- run  tests, 
do  failure  analysis,  perform  correc¬ 
tive  actions,  provide  engineering 
data. 

$1,770,000 

$1,770,000 

Contractor  A  -  $279,000  (Ref.  16) 
Contractor  B  -  $382,000 

Contractor  C  -  $659,000 

2.  Conduct  CERT  growth  test  on  two 
units  of  each  contractor 

$  230,000 

»  . 

3.  Traditional  reliability  demonstration 
and  environmental  qualification  tests 

-  - 

$1,045,000 

4.  Summary 

$2,000,000 

$2,815,000 

Program  Savings  to  Acquisition  Decision  Maker  *  41% 

q 
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TABLE  III.  BENEFITS  TO  DECISION  MAKERS  USING  A 

COMBINED  ENVIRONMENT  MISSION  PROFILE 
TEST  (CERT). 


DECISION  MAKER 

BENEFIT  TO  BE  GAINED 

Acqulsltor 

Reduce  number  of  tests 

Reduce  duration  of  testing 

Reduce  test  associated  costs 

Rapidly  improve  product 

Logistician 

Predict  sparing  needs  more  accurately 
Reduce  maintenance  manpower 

Improve  support  to  operational  user 

Operational  User 

Reduce  CND  (cannot  duplicate) 

Improve  readiness 

Reduce  maintenance  manpower 

Command  Level 

Reduce  life-cycle  cost 
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FOREWORD 


The  Air  Force  need  for  CERT  facilities  is  developed  from  an  analysis  of  the 
ongoing  and  planned  avionics  development  and  acquisition  programs.  These 
requirements  are  compared  to  the  available  Air  Force  and  Industry  CERT  facilities 
developed  from  a  government/fndustry  survey  conducted  by  TASC  to  assess  the 
future  need  for  additional  CERT  facilities.  A  listing  of  the  capabilities  of 
those  CERT  facilities  responding  to  the  survey  Is  presented.  A  recommendation 
is  made  of  how  CERT  resources  should  be  applied  and  how  additional  needs  should 
be  accommodated. 


3.7 


1.  INTRODUCTION 


The  Department  of  Defense  (DoD)  has  placed  emphasis  on  Improvement  In 
reliability  by  the  application  of  Combined  Environment  Reliability  Test  (CERT) 
to  the  detection  of  design  deficiencies,  weak  parts,  and  workmanship  defects. 

DoD  Directive  5000.40  Reliability  and  Maintainability  requires  that  "per¬ 
formance,  reliability  and  environmental  stress  testing  shall  be  combined, 
and  types  of  environmental  stress  may  be  combined  insofar  as  practical"  (Ref.  1). 
The  directive  requires  R&M  growth  during  full-scale  development,  concurrent 
development  and  production,  and  during  initial  deployment.  This  growth  is  to  be 
provided  by  a  test  and  correction  program  often  referred  to  as  Test,  Analyze  And 
Fix  (TAAF ) .  CERT  or  Mission  Profile  Testing  is  a  more  realistic  and  effective 
testing  approach  than  the  traditional  sequential  environmental  stress  testing  in 
uncovering  faults  that  occur  in  the  operational  environment. 

This  paper  addresses  the  Air  Force  CERT  testing  requirements  for  currently 
planned  avionics  programs  and  compares  them  to  the  available  Air  Force  and 
Industry  capability.  An  approach  toward  matching  resources  to  requirements  is 
suggested. 
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2.  BACKGROUND 


It  has  been  recognized  for  many  years  that  the  reliability  of  avionics 
equipment,  as  reported  for  field  operations.  Is  generally  much  lower  than  pre¬ 
dicted  and  laboratory  demonstrated  reliability.  This  has  been  attributed  to  the 
lack  of  correlation  between  the  actual  flight  environment  and  the  laboratory 
environmental  conditions  applied  during  test.  Prather  and  Earls  (Ref.  2) 
reported  the  results  of  an  Air  Force  Flight  Dynamics  Laboratory  program  which 
established  a  high  correlation  between  failures  observed  during  operations  and 
those  observed  during  a  Combined  Environment  Reliability  Test  (CERT)  con¬ 
ducted  on  a  multi -mode,  dual  channel  radar  subsystem  employed  In  a  high 
performance  aircraft.  This  success  lead  to  the  initiation  of  an  Air  Force 
CERT  Evaluation  Program  to  assess  the  technical  merit  of  combined  environment 
reliability  testing,  evaluate  the  cost  effectiveness  of  this  type  of  testing, 
and  provide  implementation  and  planning.  The  Program  Is  a  joint  effort  of 
three  AFSC  organizations  at  Wrl ght-Patterson  AFB:  Aeronautical  Systems 
Division's  PRAM  Program  Office,  ASD  Directorate  of  Engineering  and  AF  Wright 
Aeronautical  Laboratories.  The  PRAM  Program  Office  provides  overall  direction 
to  the  program.  The  AF  Flight  Dynamics  Laboratory  within  AFWAL  performs  the 
laboratory  technical  effort  In  acquiring  chambers,  formulating  test  procedures 
and  conducting  tests.  Engineering  support  to  specific  tests  Is  provided  by 
ASD  Directorate  of  Engineering  (Ref.  6). 

Since  environments  can  be  combined  in  various  combinations  and  the  inclu¬ 
sion  of  altitude  significantly  Increases  cost  without  necessarily  Improving  the 
results,  the  CERT  evaluation  program  evaluated  three  test  sequences:  full  CERT, 
CERT  without  altitude  and  MIL-STD-781C  Appendix  B  (Ref.  4).  Results  from  each 
of  these  test  sequences  were  compared  with  field  experience  to  establish  the 
relative  effectiveness  of  each  sequence.  Continued  CERT  evaluation  has  shown  a 
consistent  correlation  between  field  experience  and  full  CERT  (Ref.  5).  A  recent 
comparison  of  the  three  testing  methods  applied  to  an  IFF  system  showed  the  full 
CERT  and  CERT  without  altitude  to  produce  the  same  MTBF  results  (Ref.  6). 


The  CERT  evaluation  program  Is  nearing  Its  conclusion  with  a  final  report 
scheduled  to  be  published  in  the  near  future.  To  date,  CERT's  ability  to  stimu¬ 
late  field  failure  modes  and  rates  has  been  established  and  accepted.  The  bene¬ 
fits  of  CERT  are  more  diffuse.  It  has  been  shown  that  CERT  of  Mission  Profile 
Testing  has  benefits  to  the  Acquisition  Agent,  Logistician,  Operational  User 
and  Command  Level  (Ref.  7). 

The  Navy  has  been  using  CERT  for  monitoring  production  quality  on  captive 
carry  Air  Force  and  Navy  missiles  for  many  years.  The  results  correlate  with 
field  experience  and  are  generally  cheaper  and  quicker  than  divided  environ¬ 
ment  tests  (Ref.  8). 

Sufficient  CERT  evaluation  data  have  been  collected  to  permit  the  develop¬ 
ment  of  recomnendatlons  concerning  the  use  of  CERT  in  the  acquisition  process. 
The  DoD  CERT  Workshop  on  2-4  June  1981  Is  being  planned  to  address  this  Issue. 
This  paper  attempts  to  project  the  Air  Force  need  for  CERT  facilities  and  to 
compare  it  with  existing  Air  Force  and  Industry  CERT  capabilities. 
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3.  OVERALL  APPROACH 


To  develop  the  Air  Force  need  for  CERT  facilities,  this  paper  analyzes  the 
Air  Force's  planned  and  ongoing  development  and  production  avionics  programs 
requiring  CERT.  The  Institute  of  Environmental  Sciences  Recommended  Practices  for 
Environmental  Test  Program  Management  Is  used  as  the  basis  for  determining  the 
application  of  CERT  (Ref.  9).  Both  Air  Force  Flight  Dynamics  Laboratory  and 
Hughes  Aircraft  Company  experience  In  CERT  testing  are  used  as  the  basis  of 
facilities  utilization  (Ref.  10). 

Having  projected  the  need  for  CERT  facilities,  this  need  Is  compared  to  the 
available  resources  resulting  from  the  Industry/Government  Facilities  Survey  for 
Combined  Environment  Reliability  Testing  (CERT),  July  1980  conducted  by  TASC  for 
the  Air  Force.  Suggestions  are  made  as  to  how  the  available  CERT  resources  might 
best  be  used  and  augmented  to  meet  the  total  need. 
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3.1  Air  Force  Planned  Avionics  Development 


The  Air  Force  Avionics  Master  Plan  was  reviewed  to  Identify  programs  and 
time  phasing  for  development  and  production  (Ref.  11).  Some  39  programs  over 
five  mission  areas  were  Identified.  Tables  I  through  V  cover  the  different 
mission  areas.  The  horizontal  bars  represent  active  programs  during  the  Inter¬ 
val  of  Interest.  The  clear  bars  represent  Full-Scale  Engineering  Development 
and  the  hatch  marked  bars  represent  Production.  The  numbers  superimposed  on 
the  clear  bars  represent  the  projected  assignment  of  a  fulltime  CERT  facility. 
The  total  opposite  the  FSED  at  the  foot  of  each  column  consists  of  two  numbers 
separated  by  a  slash  mark.  The  first  number  Is  the  total  of  active  programs. 

The  second  number  Is  the  total  of  full  time  CERT  facilities.  The  total  number 
of  active  production  programs  is  on  the  bottom  line.  Those  programs  listed  as 
avionics  contained  classified  Information.  Table  VI  contains  a  summary  of 
FSED  and  Production  programs  active  during  each  year.  During  the  period  1982 
through  1986,  an  average  of  27  programs  will  require  some  form  of  CERT  testing 
including  TAAF,  Reliability  Demonstration  or  Production  Acceptance. 

3.2  Application  of  CERT  to  an  Avionics  Program 
• 

This  section  develops  an  estimate  of  the  CERT  testing  that  would  be  applied 
to  a  typical  avionics  program.  The  Institute  of  Environmental  Sciences  pro¬ 
posed  recommended  practice  on  Environmental  Test  Program  Management  Is  used  as 
a  basis  of  Identifying  the  types  of  CERT  tests  that  would  be  applied  (Ref.  9). 
The  testing  activities  which  are  the  major  contributors  to  CERT  test  duration 
are  identified  and  estimates  are  made  of  typical  test  durations.  These  test 
durations  are  used  as  a  basis  of  estimating  the  total  Air  Force  CERT  require¬ 
ments. 

Table  VII  summarizes  the  characteristics  of  environmental  testing.  The 
types  of  testing  which  are  the  major  time  consumers  are:  Test  Analyze  and  Fix 
(TAAF),  Reliability  Demonstration  (Rel  Demo)  and  Production  Acceptance.  The 
reliability  Improvement  trend  with  environmental  testing  Is  shown  In  Figure  1. 
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TAAF  and  Rel  Demo  are  part  of  Full-Scale  Engineering  Development  and 
Production  acceptance  Is  part  of  Production.  Production  Screening,  though  not 
a  large  time  consumer.  Is  a  significant  test  for  removing  "Infant  mortality" 
type  failures  and  reducing  the  number  of  field  failures.  Production  Screening 
typically  uses  combined  temperature  and  vibration  in  a  dedicated  screening 
setup. 

Each  avionics  program  will  require  a  TAAF  and  Rel  Demo  during  full-scale 
development  followed  by  Production  Acceptance  testing  during  production. 
Although  the  CERT  testing  requirements  for  each  avionics  system  are  unique  and 
would  be  tailored  to  the  specific  mission  profile,  the  approach  used  in  this 
analysis  is  to  develop  standard  testing  times  for  a  typical  avionics  system. 
Test  duration  estimates  will  now  be  developd  for  TAAF,  Rel  Demo  and  Production 
Acceptance. 


3.2.1  TAAF 


TAAF  test  durations  for  a  typical  avionics  program  are  developed  In  this 
section.  Design  guidelines  for  TAAF  programs  are  provided  In  the  Military 
Standard  for  Reliability  Growth  Testing  (Ref.  12). 

The  Duane  reliability  growth  model  Is  used  in  planning  a  TAAF  program.  An 
example  of  planning  TAAF  test  duration  is  provided  In  Figure  2.  In  this 
example,  the  predicted  MTBF  (8p)  is  1000  hours  and  the  required  MTBF  (8r)  Is 
800  hours.  A  growth  rate  (a)  of  .5  is  assumed.  The  Initial  point  of  the  cumu¬ 
lative  MTBF  line  Is  plotted  at  MTBF  *  10%  8p  or  100  hours  and  total  test  time 
of  50%  0p  or  500  hours.  The  cumulative  MTBF  line  Is  then  drawn  with  a  slope  of 
«*  .5.  The  current  MTBF  line  Is  drawn  parallel  to  the  cumulative  MTBF  line 
offset  by  1/(1  -  a  )  =2.  Where  the  current  MTBF  line  Intersects  the  required 
MTBF  line  an  ordinate  Is  dropped  to  the  horizontal.  A  total  test  time  of  8000 
hours  Is  required. 

To  assist  In  estimating  test  durations,  a  family  of  reliability  growth 
graphs  similar  to  the  one  Illustrated  in  the  example  of  Figure  2  was  deve¬ 
loped.  Two  tables  were  developed  from  the  family  of  graphs.  Table  VIII  tabu¬ 
lates  the  MTBF  growth  factor  as  a  function  of  a  for  ten  times  Increase  In  test 
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duration  (5  8p).  Table  IX  tabulates  the  test  duration  In  hours  and  multiples 
of  9p  as  a  function  of  a  for  an  eight  times  increase  in  MTBF  (0.8  8p).  If  an 
operational  MTBF  (8r)  of  200  hours  is  desired  of  a  complex  avionics  subsystem 
and  the  predicted  MTBF  (0p)  is  250  then  the  MTBF  plot  begins  at  (8p/2)  125  hour 
test  time  and  an  MTBF  (0p/lQ)  of  25  hours.  An  eight  times  growth  in  MTBF  Is 
desired.  Reference  to  Table  IX  shows  that  If  a  Is  between  .5  and  .6,  the  1250 
hour  test  duration  can  be  expected  to  produce  the  desired  200  hour  MTBF.  To 
achieve  a  400  hour  MTBF  with  a  predicted  MTBF  of  500  hours  and  an  a  between  .5 
and  .6  requires  a  2500  hour  test  duration.  Reference  to  Table  IX  shows  that 
if  a  is  .5,  the  duration  is  eight  times  the  predicted  MTBF.  TAAF  test  dura¬ 
tions  can  thus  be  expected  to  be  between  3.5  to  25  times  the  predicted  MTBF. 

For  analysis  purposes  a  factor  of  10  will  be  assumed.  This  corresponds  to  an 
a  of  .55. 

Avionics  MTBFs  can  be  expected  to  vary  from  a  low  of  25  hours  to  a  high 
of  over  1000  hours  for  simpler  subsystems.  For  purposes  of  analysis,  a  typi¬ 
cal  MTBF  requirement  of  200  hours  will  be  assumed.  Calculations  of  test 
times  will  be  based  on  this  number. 

3.2.2  Reliability  Design  Qualification  -  Rel  Demo 

Rel  Demo  test  durations  for  a  typical  avionics  program  are  developed 
in  this  section.  Design  guidelines  for  reliability  testing  for  qualifica¬ 
tion  and  acceptance  are  provided  in  MIL-STD-781C  (Ref.  13).  The  standard 
is  organized  into  test  plans  according  to  Nominal  Decision  risks  and 
Discrimination  ratio.  A  selection  of  10%  risk  and  a  Discrimination  ratio 
of  3  results  in  a  practical  risk  with  a  minimum  of  test  time.  The  minimum 
test  time  (Test  Plan  XVC  MIL-STD-781C)  is  9.3  times  the  minimum  acceptable 
MTBF.  Rel-Demo  test  times  will  run  six  to  ten  times  the  minimum  acceptable 
MTBF  to  be  demonstrated  depending  on  the  choice  of  test  and  risk.  For  pur¬ 
poses  of  analysis  a  factor  of  10  will  be  assumed. 

3.2.3  Production  Acceptance  Testing 

Production  Acceptance  test  durations  for  a  typical  avionics  program 
are  developed  in  this  section.  Production  Acceptance  testing  typically  Is 
done  on  a  sampling  basis.  The  test  plans  are  established  in  accordance  with 
MIL-STD-781C.  The  selection  of  a  test  plan  is  the  same  as  for  Rel  Demo. 

Typically,  test  times  run  six  to  ten  times  the  minimum  MTBF  to  be 
demonstrated.  For  purposes  of  analysis  a  factor  of  10  will  be  assumed. 
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3.2.4  Combined  Testing  Requirements 

This  section  combines  the  test  durations  of  the  TAAF,  Rel  Demo  and 
Production  Acceptance,  and  relates  them  to  CERT  facilities  utilization.  By 
testing  more  than  a  single  unit  in  a  test  chamber  the  calendar  test  duration 
can  be  significantly  reduced.  For  example,  two  test  samples  In  a  chamber 
reduces  the  calendar  time  by  half.  For  TAAF  and  Rel  Demo,  test  resources  are 
difficult  to  acquire;  therefore,  two  test  samples  will  be  assumed.  It  Is 
Impractical  to  have  only  one  sample  because  any  failure  disrupts  the  testing. 

Table  X  provides  test  durations  for  each  type  of  test  for  a  200  hour  MTBF 
requirement.  It  will  be  assumed  that  two  test  articles  will  be  used  as  a 
minimum  and  that  the  two  articles  will  be  tested  together  In  the  same  chamber. 

Air  Force  CERT  testing  experience  (Ref.  10,  pp.  70-71)  yields  an  average 
chamber  utilization  of  7  hours/day.  Of  that,  only  4.25  hours/day  represented  equip¬ 
ment  "ON"  time.  This  "ON"  time  Is  the  only  time  that  would  contribute  to  an 
MTBF  determination.  The  4.25  hours/day  Is  18%  utilization  of  the  test  chamber  per 
day.  Hughes'  CERT  and  MIL-STD-781B  testing  experience  (Ref.  10,  pp.  73)  pro¬ 
vides  an  average  chamber  utilization  of  1.5  hours/day  or  6.4%  utilization.  The 
industry  utilization  experience  Is  expectedly  lower  than  that  of  the  govern¬ 
ment  experience  because,  in  the  government,  case  mature  equipments  were  being 
tested.  In  the  event  of  a  failure  a  replacement  unit  was  usually  available  to 
replace  the  failed  unit.  In  the  case  of  industry  testing.  In  the  event  of  a 
failure  the  chamber  remained  idle  until  the  cause  of  failure  could  be  analyzed 
and  corrected. 

Notwithstanding  the  rationale  for  why  industry  experience  was  lower  than 
the  government's.  It  appears  reasonable  to  assume  higher  utilization  factors 
are  practicable  for  both  Industry  and  government.  It  will  be  assumed  that  both 
government  and  Industry  will  achieve  the  same  utilization  efficiency  for  each 
class  of  test  since  each  will  face  similar  delays  when  failures  occur.  Based 
on  these  assumptions,  the  test  durations  for  a  200  hour  required  MTBF  would  be 
as  shown  In  Table  XI.  A  utilization  efficiency  of  25%  Is  assumed  during 
Rel  Demo  because  the  experience  gained  during  TAAF  should  permit  more  effi¬ 
cient  utilization  of  the  test  chamber.  A  utilization  of  66%  Is  assumed  for 
Production  Acceptance  because  this  Is  a  repetitive  test  and  problems  are  typi¬ 
cally  Ironed  out  during  the  Initial  Production  Acceptance  testing.  The  66%  Is 
consistent  with  Air  Force  planning  for  CERT  testing. 
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Typically,  Rel  Demos  need  to  be  repeated  because  of  avionics  equipment 
failures,  which  would  double  the  times  shown  In  Table  XI.  A  complete  TAAF 
could  take  a  year;  however,  most  programs  depend  on  the  Rel  Demo  to  demon¬ 
strate  that  corrective  action  has  been  accomplished.  If  corrections  are 
not  incorporated  during  TAAF,  test  time  for  TAAF  can  be  reduced  but  Rel  Demo 
will  increase  in  duration.  Scheduling  of  Production  Acceptance  testing 
depends  on  production  rates.  Sufficient  units  need  to  be  produced  so  that  the 
sample  size  is  only  a  small  fraction  of  the  batch.  For  analysis  purposes,  it 


This  means  that  Production  Acceptance  testing  requires  a  chamber  two- thirds  of 


the  time. 


3.3  Projected  Air  Force  CERT  Facilities  Requirements 

This  section  develops  estimates  of  the  number  of  Air  force  avionics 
programs  that  are  in  Full-Cycle  Engineering  Development  or  Production  that 
would  require  CERT  testing.  Using  the  data  in  Tables  I  through  V  and 
assigning  a  CERT  facility  during  each  of  the  last  two  or  three  years  of  FSED 
and  a  CERT  facility  during  each  year  a  program  Is  in  production,  the  data  in 
Figure  3  were  developed.  Figure  3  provides  the  number  of  fulltime  CERT 
facilities  required  by  the  Air  Force.  Not  reflected  in  this  estimate  is  the 
condition  that  some  programs  will  have  two  contractors  doing  parallel  com¬ 
petitive  developments  during  FSED. 


3.4  Available  CERT  Facilities 

Information  on  available  facilities  was  developed  from  responses  to  the 
Industry/Government  facilities  survey  questionnaire,  discussions  with  industry 
and  government  personnel,  and  published  Information  (Ref.  14).  The 
Industry/Government  facilities  survey  was  conducted  for  the  Air  Force  by  TASC. 
Questionnaires  were  sent  to  a  selected  list  of  companies  who  were  producers  or 
users  of  CERT  facilities. 
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3.4.1  Government  CERT  Facilities 


The  Air  Force  CERT  Facilities  located  at  Wright-Patterson  AFB,  Dayton, 
Ohio  are  listed  In  Table  Xlla.  Other  Government  facilities  located  at 
Holloman  AFB,  Alamogordo,  New  Mexico  and  the  Pacific  Missile  Test  Center,  Pt. 
Mugu,  California  are  listed  in  Table  XI lb  and  c.  The  Pacific  Missile  Test 
Center  is  a  Navy  facility  which  tests  the  majority  of  the  captive  carry  tac¬ 
tical  missiles  used  jointly  by  the  Air  Force  and  Navy. 

3.4.2  Industry  CERT  Facilities 


This  section  tabulates  the  results  of  the  Industry  responses  to  the 
Industry/Government  survey  conducted  by  TASC  for  the  Air  Force.  Only  those 
industry  respondees  having  CERT  I  or  CERT  II  facilities  are  listed  in  Table 
XIII.  CERT  I  has  a  combined  environment  capability  Including  altitude  and 
CERT  II  is  the  same  without  altitude.  A  sample  of  a  complete  questionnaire  is 
provided  In  Table  Xllb.  The  tabulated  data  in  Table  XIII  represents  only 
a  portion  of  the  industry  CERT  facilities  since  a  number  of  companies  with 
extensive  CERT  experience  and  capability  did  not  respond.  All  of  the  tabu¬ 
lated  Industry  respondees  have  the  capability  to  simulate  mission  profiles. 

3.5  Matching  CERT  Resources  to  Requirements 

This  section  compares  the  Air  Force  CERT  facilities  requirements  deve¬ 
loped  in  Section  3.3  with  the  resources  identified  in  Section  3.4.  If  we 
assume  that  industry  CERT  facilities  would  be  used  for  TAAF  and  most  Rel  Demos, 
then  the  identified  requirements  from  Figure  3  range  from  2  to  19.  The 
average  requirement  for  the  period  FY82-86  is  14.  This  compares  favorably  with 
survey  results  which  In  a  partial  response  Identifies  14  companies  with  CERT 
capability. 

If  it  Is  assumed  that  the  Air  Force  will  conduct  Production  Acceptance 
testing  as  part  of  a  Production  Acceptance  Test  and  Evaluation  (PATE)  program 
then,  referring  to  Figure  3,  the  requirement  for  CERT  facilities  increases 
to  9  by  FY86.  Beginning  In  FY83,  when  the  requirement  is  for  4  CERT  faci 11- 
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ties,  the  Air  Force  capabilities  would  need  to  be  augmented.  If  the  Air  Force 
cannot  build  up  its  CERT  capabilities  in  time  to  meet  the  increased  demand,  it 
is  suggested  that  industry  facilities  be  used  for  the  testing  that  the  Air 
Force  cannot  accommodate. 

In  order  for  the  Government  to  maintain  its  expertise  in  CERT  testing,  it 
must  have  facilities  and  use  them  actively.  Since  Product  Assurance  Testing 
is  already  being  performed  by  the  Production  Acceptance  Test  and  Evaluation 
(PATE)  Department  of  the  Navy  Pacific  Missile  Test  Center  on  a  continuing  basis,  It 
would  appear  logical  to  expand  this  type  of  activity  to  cover  all  avionics 
requiring  a  PATE  type  test.  It  is  therefore  suggested  that  the  Air  Force  con¬ 
sider  undertaking  the  Product  Acceptance  Testing  and  Product  Verification 
Testing  whenever  practical  from  a  program  viewpoint.  In  addition,  the  Air 
Force  needs  CERT  facilities  for  Production  Acceptance  testing  of  avionics 
which  is  out  of  production  and  being  modified  by  Government  Depots. 

TAAF  and  Rel  Demo  should  be  performed  at  industry  facilities  under  the 
supervision  and  control  of  the  Government.  It  is  suggested  that  avionics 
developers  be  encouraged  to  develop  facilities  for  CERT  II  testing  (combined 
environments  without  altitude).  On  the  average.  It  appears  there  are  suf¬ 
ficient  CERT  II  facilities  In  industry,  however,  they  might  not  always  be  with 
the  contractor  who  is  responsible  for  the  avionics  development.  It  would  be 
ideal  if  every  avionics  system  developer  had  a  CERT  facility.  Small  avionics 
developers  could  use  independent  testing  laboratories  or  Government  facilities 
if  the  investment  in  CERT  is  beyond  their  resources. 
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4.  SUMMARY 


The  Air  Force  requirement  for  CERT  facilities  was  developed  based  on 
the  current  best  thinking  of  Government  and  Industry  environmental  testing 
specialists  and  how  CERT  should  best  be  applied  during  the  acquisition  pro¬ 
cess.  The  major  areas  for  application  of  CERT  are  TAAF,  Reliability  Design 
Qualification,  and  Production  Acceptance.  The  Air  Force  Avionics  Master  Plan 
was  analyzed  to  Identify  programs  that  were  in  full  scale  development  and/or 
production  to  determine  CERT  requirements  in  a  time  phased,  program  by  program 
basis.  The  analysis  resulted  in  top  level  summary  of  Air  Force  CERT  needs  by 
year.  These  requirements  were  compared  to  available  Government  and  Industry 
CERT  Facility  resources.  Based  on  a  limited  response  survey,  it  appears  that 
industry  can  handle  the  current  CERT  needs.  The  Air  Force  needs  to  Increase 
the  number  of  CERT  facilities  for  Production  Acceptance  testing  on  out-of- 
production  avionics  modifications. 
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TABLE  I.  AVIONICS  PROGRAM  AIR-TO-SURFACE  ATTACK 


TABLE  II.  AVIONICS  PROGRAM  COMMAND,  CONTROL  AND  COMMUNICATIONS 
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TABLE  III.  AVIONICS  PROGRAM  COUNTERAIR 
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TABLE  VI.  AVIONICS  PROGRAM  SUMMARY 
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TABLE  VII.  SUMMARIZED  ENVIRONMENTAL  TEST  CHARACTERISTICS 
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Figure  1.  Reliability  Improvement  Trend 
Environmental  Testing 
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TOTAL  TEST  TIME  (HOURS) 


Figure  2.  Example  of  Planning  TAAF  Test  Duration 


TABLE  VIII.  MTBF  GROWTH  FOR  A  TEN  TIMES  INCREASE  IN  TEST  DURATION 
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TABLE  X.  TEST  TIMES  FOR  200  HR  MTBF  REQUIREMENT 


TYPE  TEST 

TEST  DURATION 

TAAF 

2000  HR 

REL  DEMO 

2000  HR 

PRODUCTION  ASSURANCE 

2000  HR 

TABLE  XI.  CALENDAR  TEST  DURATIONS  FOR  200  HR  MTBF  REQUIREMENT 

2  units/chainber 
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CERT  FACILITIES  REQUIREMENTS 
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Figure  3.  Air  Force  CERT  Facilities  Requirement 
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TABLE  XII.  GOVERNMENT  CERT  FACILITIES 
a)  Air  Force  Wright  Aeronautical  Laboratories 


CEBT  I 

Chamber : 

Size:  4  ft  diameter  \  5>  it  long  (1.2  m  dia.  *  1.5  ml 

Range  Change  Rale 

Altitude  0  to  66.500  ft  (20. 5  kmi  ±58,000  ft /min 

(±1 1.6  kro  'mini 

Temperature  — 100*  to  +  500*1  ±9*F/mm 

(—75*  to  149*0  (i5*f  mini 

Humidity :  0  0005  to  0  04  U  «  ±0.04  Hsroin 

(\Vc  —  lb  HzO  lb  dry  am 

ECS  Cooling  Air: 

Mass  Flow  0  to  5.7  Ib'min  (2  6  kg/min)  ±1/0  lb  min  mm 

(±77.5  kg/rnin/mtn) 

T  em[>erature  —90*  to  -f  200*F  ±100*F'min 

(—68*  to  95*0  ( ±56’C.  mini 

Humidity  0.0005  to  0.04  Ws  ±0.04  \\>  'min 

Vibration : 

Simulator:  one  electromagnetic  shaker 
Frequence  5  to  2,000  Hr 
Modes:  sinusoidal  or  random 

Limits:  1  inch  (2.5  cm)  P-P  amplitude.  8.000  lb  force 

(5650  kg  force  1 

CERT  II 

Chamber : 

Size:  7  fi  diameter  x  10  ft  long  (2.1  m  dia.  x  5  ml 
Range  Change  Rale 

Altitude:  0  to  70.000  ft  (21.4  km)  +10,000  ft/mtn 

(5  km  /min)  ascending 

—20.000  ft/min 

(6.1  km /min)  descending 

Airflows  two  independence  controlled:  one  for  bav  emimnmrnt. 
one  for  F.C.S  moling  air 
Airflow  Capabilities  (each  flow): 

Range  hangr  Rale 

Mass  Flow:  0  to  7.5  lb'  mm  ±100  lb  min'mm 

(5.4  kg'min)  ±(45.4  kg'minmtii 

Temiteraitire:  —65*  to—  lfln'F  ±l20*F'min 

(  —  54*  to  7l*(  )  (±ri7*C./min) 

Htimtdits*  0.00001  to  0.04  \V-  ±0.0?  Ws'min 

Vibration : 

Simiil.tioi  one  clrrtromagnrtit  shake) 

Frequent'  5  to  2.000  Mr 
Modes  sinusoidal  or  random 

Limits  1  ins li  (2  5  etui  IM’  amplitude.  17.600  ||>  fonc 

(7950  kg  forir1 

CERT  III 

Chamber : 

Size:  5  ft  wide  x  5  ft  high  x  5  It  deep  (1.5  v  1.5  x  1.5  nn 
(inside  dimensions) 

Range  C  hange  Role 

Altitude:  0  to  70,000  fi  (21.4  km)  -‘-10.000  ft 'min 

(5  km  mini  ascending 

—20  OoO  fi  min 

(6.1  km  nnn  1  descending 

Airflows:  two  independent!)  controlled:  one  for  fu'  <  n>  uonmciit 
one  for  ILLS  cooling  an 
Airflow  Capabilities  (each  flow): 

Range  (  hange  Rale 

Mass  Flow:  0  to  7.5  lb/min  ±100  lb  min'nnn 

<5.4  kg/min)  ±45  4  kg  mm  mu. 

Temperature  — 65’  to  +I60’F  ±120*i  mm 

(—54*  to  71*0  (±67*(.'mtni 

Humidits'  0.00001  to  0.04  \V-  ±0  04  5Vs  mm 

Vibration : 

Simulator  one  electromagnet  it  sbakei 
Frequents  5  to  2,000  H/ 

Modes  sinusoidal  or  random 

Limns.  1  inch  (2  5  cm)  I’-P  amplitude.  17.500  lb  font 

-  (7950  kg  fort  i 

’Humidits  control  is  fot  its  tooling  an  only _ _ 
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TABLE  XII.  GOVERNMENT  CERT  FACILITIES  (Continued) 
b)  Central  Inertial  Guidance  Test  Facilities,  Holloman  AFB 


CHAMBER 

CHAMBER 

1.  Chamber  Dimensions 

4,x4'x4' 

8'x8'xn ' 

2.  Test  Specimen 

Max.  Dia. 

Max.  Wt. 

3.  Simulated  Environments 

a.  Temperature 

Range  °F 

Rate  of  Chg.  °F/min 

-100  +  300 

13.5 

-100  +  200 

1 

b.  Vibration 

Random  lbf-Hz 

Sinusoidal  lbf-Hz 

No.  Axis  &  Orient. 

Max.  Specimen  Wt 

9000  # 
3-5000 

9000  # 
3-5000 

1 

17,000  if 
3-3000 

17,000  if 
3-3000 

1 

Range  ft 

Rate  of  Climb  ft/min 

Rate  of  Descent  ft/min 

100,000 

7,000 

7,000 

200,000 

35,000 

35,000 

d.  Humidity 

Range  %  RH 

Rate  of  Change 
%  RH/min 

30-95 

e.  Cooling  Air 

Capacity  Ib/min 

Rate  of  Change 
lb/min/min 

Range  °F 

Rate  °F/min 

%  RH  -  Range 

%  RH/min 

** 

A 

-40  to  158 

13.5 

30-95 

6.5 

4.  What  is  your  ability  to  simulate  mission  profiles?  Simulates  mission 
profiles  with  HP  Digital  Control  System 

5.  What  environments  can  be  simultaneously  and  dynamically  controlled? 
a,  b,  c,  d,  e 

4°C  and  above 


Available  after  1  January  1982 
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TABLE  XII.  GOVERNMENT  CERT  FACILITIES  (Continued) 
c)  CERT  Facilities  at  the  Pacific  Missile  Test  Center  (Navy) 
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a  The  most  recent  description  of  the  facilities  and  test  results  is  provided  by  D  B  Meeker  and 
w  p  Everetr.  US  Maw  Experience  on  the  Effects  of  Carrier  Aircraft  Environment  on  Guided 
Missiles,  aGaRD  conference  proceedings  Number  CP270.  Mav  1979. 

n  The  vibration  technique  in  chis  facility  is  described  by  C.V.  Ryden .  "Dual  Shatter  Vibration 
Facility,  Shock  and  Vibration  Bulletin,  dumber  46,  Part  3,  (1976),  pp  27-53 

z  The  effectiveness  of  this  acoustic  facility  is  described  by  J.C.  Calkins  and  A  C  Piersol. 

Simulation  of  the  SPARROW  Vibration  Environment,''  SAE  Booklet.  Number  '30939.  '1973) 

i  The  combined  thermo-acoustic  facility  was  first  described  by  V.D.  Everett.  "Thermo-Acoustic 

Simulation  of  Captive  Flight  Environment.'*  Shock  and  Vibration  Bulletin.  Number  **6.  Part  3  (197b) 
PP  103-111. 

«  The  rationale  for  the  transition  to  this  facility  by  a  M  Spandno  snd  M  L  Burke.  Acoustics  or 
Shatters  for  Simulation  of  Captive  Flight  ViDration.'*  Shock  and  Vibration  Bulletin.  Number  -6 . 

Part  9  .  (  1978)  .  pp  5-13. 
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FOREWORD 


The  use  of  Combined  Environment  Reliability  Testing  Is  discussed  as  a 
tool  for  achieving  product  reliability.  The  selection  of  the  appropriate 
test  strategy  to  fulfill  multiple  test  objectives  is  discussed  as  well  as 
how  the  selection  is  affected  when  dealing  with  equipments  destined  for 
multiple  aircraft  applications. 


1 .  BACKGROUND 


During  the  last  decade  or  so,  a  number  of  efforts  have  been  undertaken  to 
improve  the  methods  for  testing  and  evaluating  avionics.  Some  of  Col.  Ben 
Swett's  efforts  and  the  report  to  the  Air  Force  Studies  8oard  on  Reliability  in 
Aeronautical  Avionic*?  Equipment  in  1975,  for  example,  were  quite  well  publicized 
These  efforts  were  generally  critical  of  the  lack  of  realism  in  the  reliability 
testing  process  for  avionics  equipment. 

At  about  the  same  time,  there  was  a  growing  awareness  that,  if  the  ratio  of 
dollars  for  OSS  (Operations  and  Support)  to  dollars  for  new  equipment  continued 
to  increase,  almost  all  available  funds  would  be  consumed  by  OSS  before  too  ma'\v 
year:  elapsed.  At  that  point,  resources  would  be  unavailable  for  investment  i- 
new  equipment  required  to  meet  changing  threacs.  Thus,  the  PRAM  (Productivity, 
Reliability,  Availability  and  Maintainability)  Program  Office  was  chartered  in 
1975  to  mount  a  concerted  and  focused  effort  on  developeing  methods  to  reduce 
the  Air  Force's  overall  OSS  burden. 

lust  as  PRAM  was  being  formed,  the  Air  Force  Flight  Dynamics  Laboratory 
( AFFDL )  was  finishing  an  experimental  CERT  (Combined  Environment  Reliability 
Testing)  evaluation  of  the  F-lll  Terrain  Following  Radar  and  concluded  that  the 
laboratory  results  correlated  well  enough  with  field  experience  to  merit  further 
investigation  of  CERT.  These  activities  provided  impetus  to  incorporating  some 
form  of  CERT  within  the  version  of  MIL -5TD-781C  which  was  then  in  a  coordination 
cycle.  Thus,  the  scene  was  set  for  PRAM,  along  with  AFFDL  and  Aeronautical 
System  Division's  Deputy  for  Engineering  (ASD/EN),  to  embark  on  an  evaluation  of 
the  technical  merit  and  cost-effectiveness  of  CERT.  On  the  one  hand  was  the 
possibility  of  duplicating  and  then  fixing  every  possible  failure  that  might  be 
encountered  in  the  field  being  offset  by  potentially  very  large  expenditures  for 
facil iti zation,  testing,  and  incorporating  fixes. 

As  is  well  known  and  documented,  the  CERT  evaluation  has  been  in  progress 
since  then  and  should  be  concluded  this  year.  During  this  period,  though,  it 
became  evident  that  other  variables,  besides  test  conditions  and  field  reliabi¬ 
lity.  were  driving  OSS  costs.  Another  major  influence  toward  increasing  OSS 


costs  was  the  proliferation  of  avionics,  along  with  the  increasingly  expensive, 
dedicated  and  often  peculiar  support  systems  required  to  keep  the  avionics 
operational.  To  address  the  proliferation  problems  and  promote  standardization, 
the  Air  Force  established  the  Deputy  for  Avionics  Control  in  1978. 

A  major  function  of  the  Deputy  for  Avionics  Control  is  to  serve  as  secre¬ 
tariat  for  the  Air  Force’s  annual  Armament  and  Avionics  Planning  Conference  and 
to  produce  avionics  planning  guidance  for  Headquarters  USAF  as  a  result  of  the 
conference.  The  conference  consists  of  four  standing  panels:  Availability, 
Computer  Resources,  Combat  Effectiveness,  and  Standardization.  The  following 
list  represents  a  sample  of  the  avionics  that  the  Standardization  Panel  has 
endorsed  as  future  "standards"  to  be  installed  in  “multiple-host"  aircraft: 

a.  CARA  -  Combined  Altitude  Radar  Altimeter 

b.  MRR  -  Multi-Role  Radar 

c.  CADC  -  Central  Air  Data  Computer 

d.  ARC- 164  Replacement 

e.  Standard  Crash  Data  Recorder. 

Obviously,  changes  can  and  probably  will  occur.  The  point  is  that  more  and 
more  standard  avionics  are  programmed  and  destined  for  our  operational 
i nventories . 

Some  recent  acquisition  strategies,  such  as  concurrency  (entering  produc¬ 
tion  before  completing  Full-Scale  Engineering  Development)  have  served  to  high¬ 
light  the  role,  importance  and  character  of  developmental  testing  as  a  tool  in 
achieving  product  reliability.  Since  concurrency  tends  to  disallow  programmatic 
windows  for  formal  reliability  qualification  tests,  TAF  ( Test-Anal yze-Fix)  is 
emerging  as  one  of  the  more  important  tools  for  Full-Scale  Engineering 
Development  application.  And,  as  CERT  is  maturing,  it  is  becoming  an  integral 
element  of  the  TAF  process,  in  addition  to  its  utility  during  other  phases  of  a 
reliability  program.  These  considerations  are  embodied  in  the  recent  DoD 
Directive  5000.40  which  requires  that: 
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"...  performance,  reliability  and  environmental  stress 
testing  shall  be  combined,  and  types  of  environmental 
stress  will  be  combined  insofar  as  practical..." 

with  an  emphasis  on  reliability  and  maintainability  growth. 

In  view  of  increasing  standardization  and  the  integration  of  CERT  and  TAF 
techniques  as  influenced  by  acquisition  strategies  such  as  concurrency,  this 
paper  is  intended  to  stimulate  workshop  discussion  on  this  issue: 

For  systems  destined  for  multiple-host  applications, 
how  are  CERT  tests  most  effectively  structured? 

Some  concepts  employed  in  the  past  and  other  ideas  are  provided  in  the  sub¬ 
sequent  sections  to  establish  the  discussion  baseline. 

2.  CURRENT  REQUIREMENTS 

A  wealth  of  information  on  existing  testing  requirements  can  be  found  in  the 
various  directives,  regulations  and  Military  Standards.  Both  MIL-STD-781C  and 
MIL-STD-785B  describe  the  need  for  Integrated  Testing,  with  785B  language  clo¬ 
sely  reflecting  the  wording  of  DoO  Directive  5000.40. 

MIL-STD-781C  lists  the  following  tests  in  its  breakout  discussion  of 
integrated  testing: 

a.  Reliability  development  tests 

b.  Environmental  development  tests 

c.  Environmental  qualification  tests 

d.  Reliability  qualification  tests 

e.  Reliability  acceptance  tests 

f.  Burn-in  and  screening  tests. 

The  new  MIL-STD-785B  lists  the  same  tests  and  characterizes  them  in  terms  of 
reliability  engineering  tests  and  reliability  accounting  tests.  In  general ,  the 
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integrated  test  programs  progress  from  environmental  testing  (MIL-STD-810 )  to 
reliability  development  growth  testing  and  then  to  the  “accounting"  reliability 
qualification  and  production  reliability  acceptance  tests. 


MIL-STD-781C  contains  ample  guidance  with  respect  to  developing  testing  en¬ 
vironments  as  a  function  of  composite  mission  profiles.  Most  past  testing  has 
probably  been  accomplished  using  the  composite  environmental  profiles  relevant 
to  a  single  weapon  system  for  which  the  avionics  has  been  targeted. 

Now,  if  the  implications  of  standardization  are  considered  -  -  a  composite 
mission  profile  takes  on  a  different  meaning  for,  in  their  case,  the  composite 
should  account  for  transport,  bomber  and  fighter  missions,  as  contrasted  with  a 
composite  within  one  MDS  (Mission  Design  Series)  aircraft.  Again,  DoD  Directive 
5000.40  stipulates  that  "Test  conditions  and  procedures  shall  be  operationally 
realistic..."  In  summary,  the  policy  as  well  as  the  technical  guidance  are  in 
place.  The  remaining  challenge  is  in  developing  a  conceptual  framework  for 
structuring  tests  that  can  minimize  life-cycle  cost  for  standard  avionics  across 
multiple-host  platforms. 

3.  EXISTING  OPTIONS  FOR  MEETING  REQUIREMENTS 


A  major  problem  which  confronts  government  organizations  responsible  for 
reliability  programs  is  the  selection  of  tests  which  can  aid  in  attaining 
program  reliability  requirements.  Schedule  and  funding  constraints  mandate  a 
cost-effective  selection.  The  reliability  test  program  should  serve  three 
objectives: 


a.  Disclose  deficiencies  in  designs,  material 
and  workmanship. 

b.  Provide  data  to  support  manpower  and  logistics 
support  cost  estimates. 

c.  Determine  compliance  with  quantitative 
reliability  requirements. 
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Several  options  are  discussed  in  the  following  sections. 


3.1  Serial  Testing 

Serial  testing,  in  the  context  of  this  report,  is  comprised  of  a  series  of 
tests  which  are  structured  to  meet  the  reliability  test  objectives  stated  above. 
Several  types  of  testing  can  be  used  complementarily  in  order  to  meet  stated 
objectives.  A  frequent  example  is  that  of  MIL-STD-810  type  environmental  tests 
followed  by  Reliability  Qualification  Testing  (RQT).  The  remainder  of  this  sec¬ 
tion  addresses  these  and  other  types  of  testing. 

Environmental  tests  are  specifically  designed  to  disclose  weak  parts  and 
workmanship  defects  for  correction.  These  tests  should  be  applied  to  parts, 
components,  subassemblies,  assemblies  or  equipment  to  remove  defects  which 
would  otherwise  cause  failures  during  early  field  service.  The  test  conditions 
and  procedures  should  be  designed  to  stimulate  failures  typical  of  field  ser¬ 
vice,  rather  than  provide  precise  simulation  of  operational  life  profile. 
Environmental  stress  types  may  be  applied  in  series  rather  than  in  combination 
and  should  be  tailored  for  the  level  of  the  assembly  at  which  they  are  most  cost- 
effective. 

Engineering  Development  testing  using  CERT  (Combined  Environment 
Reliability  Testing)  is  a  strategy  that  can  result  in  early  identification  and 
correction  of  design  deficiencies.  If  CERT  were  used,  for  example,  in  a 
planned,  pre-qualification,  test-analyze-fix  (TAF)  process,  real  reliability 
growth  is  possible.  A  key  point  here,  however,  regardless  of  test  methodology, 
is  that  appropriate  and  selective  corrective  action,  taken  to  preclude  opera¬ 
tional  repetition  of  test-induced  incidents,  is  the  mechanism  that  actually 
improves  reliability.  If  employed,  it  is  imperative  that  TAF  testing  for  the 
sake  of  reliability  growth  be  implemented  early  in  Engineering  Development 
because  the  early  iterative  design  process  is  more  fluid,  more  accommodating  of 
design  changes,  and  less  subject  to  formal  configuration  control.  The  later 
that  TAF  using  CERT  is  initiated,  the  greater  are  the  risks  of  postponing 
required  corrective  actions  and  incurring  later  retrofit  problems. 
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Reliability  Qualification  Testing  is  intended  to  provide  the  government 
some  assurance  that  minimum  acceptable  reliability  requirements  have  been  met 
before  items  are  committed  to  production.  RQT  should  reflect  operational  con¬ 
ditions  and  permit  estimates  of  demonstrated  reliability.  The  statistical  test 
plan  must  pre-define  criteria  of  compliance  which  limit  the  probability  that  true 
reliability  of  the  item  is  less  than  the  minimum  acceptable  reliability  require¬ 
ment.  RQT  is  required  for  items  that  are  newly  designed,  for  items  that  have 
undergone  major  modifications,  and  for  items  that  have  not  met  their  allocated 
reliability  requirements. 

In  sumnary,  with  schedules  and  resources  permitting,  an  appropriate 
sequence  of  reliability  testing  would  be:  (1)  environmental  tests  to  remove 
defects  in  the  test  items,  (2)  Engineering  Development  tests,  perhaps  employing 
CERT  methods  in  a  TAF  process,  to  disclose  and  correct  design  deficiencies  and 
defects,  and  (3)  RQT,  again  using  CERT  methods,  to  provide  reasonable  assurances 
of  having  met  minimum  acceptable  reliability  requirements. 

3.2  Simultaneous  Testing 

As  discussed  in  the  above  section,  three  types  of  reliability  tests  have 
been  identified  which  together  serve  the  reliability  test  objectives.  However, 
schedules  are  not  always  adequate  to  allow  for  series  testing  In  which  infor¬ 
mation  learned  from  one  test  series  may  be  passed  to  the  next  test  series.  For 
this  reason,  the  government  utilizes  simultaneous  testing,  i.  e.,  combining  two 
or  more  test  objectives  at  the  same  time.  In  a  test  program  constrained  by 
schedule  or  budget,  it  may  be  desirable  to  Incorporate  selected  qualification 
test  levels  in  a  combined  environmental  test.  The  incorporation  of  qualifica¬ 
tion  test  levels  In  the  test  profile  may  Involve  exposing  those  units  under  test 
to  levels  of  stress  which  exceed  those  under  which  they  are  designed  to  operate; 
however,  evaluation  of  the  equipment  can  be  enhanced  by  the  accumulation  of  data 
relating  to: 

a.  Equipment  environmental  limits 

b.  Early  discovery  of  latent  failure  modes 

c.  Marginal ity  of  design. 


3.8.7 


Sensitivity  of  equipment  designs  to  severe  environments  should  become  readily 
apparent.  Failure  modes  that  would  not  ordinarily  be  observed  for  thousands  of 
test  hours  or  several  months  in  the  field  may  be  “shaken  out"  early.  In  a  com¬ 
petitive  procurement,  the  relative  sensitivities  of  different  contractors' 
equipments  to  stress  levels  above  the  design  limits  may  be  of  Importance  to  the 
government  as  a  proposed  evaluation  criterion.  The  greatest  benefit  to  the 
government,  however,  is  the  potential  for  design  improvements  which  are  proposed 
or  implemented  by  the  contractor  on  data  obtained  while  exposing  the  equipment 
to  combined  environmental  stresses. 

3.3  Combined  Environment  Reliability  Test  Objectives 


As  alluded  to  in  the  previous  section,  it  is  often  necessary  to  blend  test 
objectives  if  they  are  to  be  met  within  schedule  and  budget  constraints.  While 
it  may  be  impossible  to  meet  all  test  objectives  through  a  single  test  series, 
multiple  objectives  across  test  series  can  be  accommodated  by  applying  dif¬ 
fering  versions  of  CERT.  The  combined  simulated  environments,  characteristic 
of  CERT  as  used  in  an  RQT,  for  example,  can  be  enhanced  or  intensified  and  used 
as  a  stress  screen  to  stimulate  additional  failures  or  incidents  that  become 
corrective  action  candidates.  This  procedure,  when  integrated  with  dedicated 
failure  analysis  and  corrective  action  systems,  is  referred  to  as  the  TAF  metho 
dology. 

CERT  is  a  powerful  test  that  can  be  utilized  for  various  purposes.  By 
exposing  equipment  to  typical  (simulated)  or  worst-case  (stimulative)  flight 
environments  in  the  laboratory,  the  following  objectives  of  CERT  may  be 
fulfilled: 


a.  Failure  mode  Isolation 

b.  Equipment  qual ification 

c.  Accelerated  testing 

d.  Reliability  prediction 

e.  Reliability  growth. 
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Not  all  of  these  objectives,  however,  can  be  accomplished  in  a  single,  combined 
environment  test.  The  realization  of  each  depends  on  the  characteristics  of  the 
selected  environment  profiles  and  on  the  specific  test  methodology  employed. 
Besides  the  realization  of  failure  mode  isolation  or  equipment  qualification 
which  have  previously  been  discussed,  the  objectives  which  support  the  three 
reliability  test  program  types  listed  earlier,  are  discussed  in  the  following 
paragraphs. 

Accelerated  Testing  -  The  greater  the  time  and  the  amount  of  data  accumu¬ 
lated,  the  greater  the  confidence  that  can  be  placed  in  the  results  from  the 
tests,  especially  the  reliability  estimates.  With  combined  environment  testing 
using  acceleration  techniques,  it  is  possible  to  accumulate  a  large  amount  of 
simulated  flight  data  in  a  relatively  short  period  of  time.  There  are  two 
important  types  of  test  acceleration: 

a.  Time  acceleration 

b.  Environment  acceleration. 

In  a  time-accelerated  test,  the  simulated  environment  profiles  deemphasize  the 
benign  mission  phases  that  are  assumed  not  to  generate  many  failures.  The 
benign  phases  include  extended  cruise  and  ground  time.  Accordingly,  most  of  the 
test  cycle  Is  devoted  to  the  more  stringent  environments  associated  with 
takeoff,  climbout  and  altitude  changes.  If  the  assumption  is  valid  that  few 
failures  are  induced  by  the  benign  mission  phases,  then  the  ratio  between  the 
times  of  the  average  actual  mission  and  the  laboratory  mission  is  a  valid  test 
acceleration  factor  (k-factor)  applied  to  laboratory  MTBF  In  order  to  estimate 
field  MTBF.  This  factor  is  used  to  convert  test  time  to  equivalent  flight  time 
for  field  reliability  prediction.  The  k-factor  is  clearly  laboratory  test- 
mission  dependent. 

In  an  envi ronmental ly-accel era ted  test,  the  environment  levels  employed  are 
more  severe  than  those  associated  with  a  typical  mission  profile  (or  with  the 
design  limits  of  the  equipment).  Accelerated  environments  are  used  to  stimulate 
and  detect  equipment  failure  modes  that  would  not  be  exhibited  under  expected 
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flight  conditions  or  would  not  be  observed  until  the  equipment  was  exposed  to 
normal  conditions  for  longer  periods  of  time  associatec)  with  equipment  wear-out 
modes.  It  is  difficult,  for  test  environments  accelerated  beyond  normal  flight 
levels,  to  derive  a  test  acceleration  factor  for  converting  test  results  to 
equivalent  flight  results. 

Reliability  Estimation  -  in  a  competitive  procurement,  such  as  the  OMEGA 
Program  2041  in  which  equipment  life-cycle  cost  was  a  primary  evaluation  cri¬ 
terion,  the  life-cycle  cost  of  competing  contractor's  equipment  may  exhibit  dif¬ 
ferent  sensitivities  to  variations  in  equipment  MTBF.*  For  these  procurements, 
consistent  reliability  estimation  is  an  important  issue  and  combined  environment 
testing  may  be  a  primary  source  for  such  reliability  data. 

The  combined  environment  test  methodology  was  developed  because  of  the  ina¬ 
bility  of  traditional  test  methods  to  adequately  predict  the  nature  of  field 
behavior  for  avionics  equipment  prior  to  deployment.  The  major  problems  with  the 
mil-STD-781B  test  methodology  are: 

a.  Absence  of  a  realistic  field  environment  in  the  laboratory. 

b.  Lack  of  acceptable  levels  :f  correlation  between  laboratory 
and  field  reliability. 

n*  „>*•  -if  the  dynamic  combination  of  temperature,  pressure,  humidity  and  vibra¬ 
tion  n  th*>  laboratory  test  flight  pofile  is  a  major  development  in  overcoming 
‘  »<■  '  •  X  sta-  1  e . 

>1  addressing  the  laboratory/fiel d  reliability  correlation  problem,  it  is 
important  to  distinguish  between  two  types  of  combined  environmental  tests: 

a.  Combined  environmental  rel iabil ity  test. 

b.  Combined  environmental  stress  test. 

*  It  is  possible,  given  one  set  of  MTBFs  for  the  various  competing  contractors, 
that  the  ranking  by  life-cycle  cost  would  change  if  the  MTRF  for  each  contractor 
were  doubled  or  changed  by  a  similar  factor. 


If  the  test  objective  is  to  estimate  the  equipment's  field  reliability,  then  the 
environmental  levels  in  the  laboratory  should  be  compatible  with  those  in  the 
field  to  simulate  failure  modes.  If  the  objective  of  the  test  is  to  identify 
equipment  environment  sensitivity  and  tolerance,  then  accelerated  environments 
should  be  employed  to  stimulate  failure  modes. 

Reliability  Growth  -  There  are  two  types  of  reliability  related  issues: 

a.  Reliability  estimation 

b.  Reliability  growth. 

As  discussed  above,  valid  reliability  prediction  requires  the  selection  of  a 
representative  environment  profile  and  the  derivation  of  an  appropriate  correla¬ 
tion  factor  to  relate  laboratory  test  experience  to  expected  operational 
experience.  Testing  to  achieve  reliability  growth  may  not  be  compatible  with 
testing  to  predict  field  reliability.  For  example,  in  a  program  constrained  by 
schedule,  "locally  optimum"  reliability  growth  may  require  stressing  the  equip¬ 
ment  to  its  limits,  thereby  uncovering  failure  modes  which  result  in  early 
design  improvements.  Such  an  approach  may  reduce  confidence  in  field  reliabi¬ 
lity  estimates  due  to  problems  associated  with: 

a.  Deriving  a  k- factor  for  accelerated  environments 

b.  Allocating  test  data  among  different  design 
configurations  of  the  same  equipment. 

Therefore,  during  the  planning  stages  of  a  test  program,  the  engineer  must  be 
prepared  to: 


a.  Evaluate  cost-effectiveness  tradeoffs 
between  reliability  growth  and  reliability 
prediction. 

b.  Develop  guidelines  for  matching  environ¬ 
ment  stress  levels  with  the  available 
program  test  time. 

( 
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3.4  Test  Application 

The  government  may  wish  to  apply  any  or  all  of  the  above  referenced 
reliability  tests  on  selected  weapons  systems.  When  the  avionics  system  is 
being  procured  or  tested  for  a  single-aircraft  application,  the  problem  of  iden¬ 
tifying  the  range  and  duration  of  environments  within  which  the  equipment  is 
expected  to  operate  is  simpler  than  the  case  of  a  multiple-host  application. 
After  determining  the  environmental  conditions  representative  of  those  encoun¬ 
tered  by  the  equipment  aboard  the  host  aircraft  during  a  typical  mission,  the 
selected  environment  profile  should  cover  the  major  operating  areas  of  the  host 
aircraft  flight  profiles.  Representative  profiles  can  be  developed  by  uti¬ 
lizing  all  available  data  from  flight  logs  and  historical  records. 

The  test  objectives  for  avionics  destined  for  multiple  applications  should 
remain  relatively  unchanged.  However,  the  task  of  selecting  the  appropriate 
test  environment  for  a  multiple-host  implication  can  take  on  added  dimensions 
in  terms  of  selecting  appropriate  test  condniur.i  that  will  minimize  operating 
and  support  (OSS)  costs.  It  is  not  entirely  clear  that  testing  a  piece  of 
avionics  equipment  under  the  most  severe  envircr~ental  conditions  that  it  might 
encounter  on  one  of  its  host  aircraft  would  be  most  effective.  In  the  case  of 
a  homogeneous  population  of  avionics  equipment,  it  Is  readily  conceivable  that 
equipment  currently  used  in  a  fighter  environment  may  later  be  installed  and 
operated  in  bomber  or  turboprop  environments. 

Assuming  some  sort  of  limits  on  test  time  and  resources,  the  temperature 
(also  vibration,  humidity,  etc.)  profiles  should  be  chosen  so  as  to  result  in 
highest  life-cycle  payoffs.  It  might  be  judicious  to  test  to  environments  that 
are  not  necessarily  the  most  severe,  but  that  will  be  most  common  In  order  to 
effect  the  maximum  reduction  of  O&S  costs.  The  concept  of  a  composite  profile, 
i.e.,  across  applications,  for  each  environment  also  merits  analysis.  Figure  1 
which  can  be  interpreted  as  a  discrete  distribution  of  temperatures,  actually 
depicts  the  temperature  profiles  from  MIL-STD-781C  for  jet  and  turboprop 
aircraft.  Temperature  has  been  plotted  for  10  aircraft  conditions  across  14 
mission  phases.  Each  aircraft  condition  and  mission  phase  is  specified  in 
Table  I.  As  can  be  seen  in  Figure  1,  temperature  for  the  first  5  aircraft 
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INDEX 

MISSION  PHASE 

AIRCRAFT  TYPE 

1 

Ground  Runup 

Fighter  Air-Conditioned  Cold 

2 

Takeoff 

Bomber  Air-Conditioned  Cold 

3 

Climb 

Turboprop  Cold 

4 

Cruise  High 

Fighter  RAM  Cold 

5 

Cruise  Medium 

Bomber  RAM  Cold 

6 

Cruise  Low 

Fighter  Air-Conditioned  Hot 

7 

Combat  Low 

Bomber  Air-Conditioned  Hot 

8 

Combat  Medium  Low 

Turboprop  Hot 

9 

Combat  Medium  High  (L) 

Fighter  RAM  Hot 

10 

Combat  Medium  High  (H) 

Bomber  RAM  Hot 

11 

Combat  High 

Not  Used 

12 

Start  Descent 

Not  Used 

13 

End  Descent 

Not  Used 

14 

Land 

Not  Used 
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types  (which  are  operating  in  a  cold  environment)  range  from  -54°C  to  -26°C  for 
the  bomber/air-conditioned  (aircraft  type  2)  to  -55°C  to  19°C  for  the 
fighter/RAM  air  cooled  (aircraft  type  4).  Likewise,  temperature  for  the  last  5 
aircraft  types  (which  are  operating  in  a  hot  environment)  range  from  -28°C  to 
71°C  for  the  bomber/RAM  air  cooled  (aircraft  type  10)  to  10°C  to  93°C  for  the 
fighter/air-conditioned  (aircraft  type  6).  Simple  comparison  shows  that  the 
fighter/RAM  air  cooled  covers  a  wider  temperature  region  for  the  cold  environ¬ 
ment  while  the  bomber/RAM  air  cooled  provides  a  wider  temperature  region  for  the 
hot  environment  with  the  fighter/air-conditioned  category  containing  the 
highest  peak  temperature. 

Thus,  the  temperature  environment  from  MIL-STD-781C  is  most  severe  for 
fighters,  a  condition  probably  magnified  when  the  time  dimension  is  factored  in. 
A  typical  fighter  mission  may  last  approximately  1.5  hours  while  a  bomber 
mission  may  exceed  10  hours.  When  one  considers  that  the  temperature  ranges  for 
the  fighter  aircraft  must  be  covered  more  quickly  than  for  a  bomber  aircraft,  it 
is  apparent  that  the  rate  of  change  in  temperature  is  more  rapid  and  probably 
more  stressful  in  the  fighter. 

If  the  solution  to  testing  is  to  select  the  worst  environment,  then  the 
fighter  environment  should  be  selected.  However,  suppose  that  the  equipment 
selected  is  to  be  installed  in  a  limited  number  of  fighters,  and  in  many  bomber 
and  cargo  aircraft.  If  the  fighter  environment  Is  used,  what  should  be  done 
with  failure  models  peculiar  to  the  extremes  of  the  hot  temperature  profile 
(i.e.,  93°C )  which  will  not  be  experienced  by  the  bomber  and  cargo  aircraft? 

Even  more  important,  what  is  the  proper  disposition  of  failures  resulting  from 
rapid  temperature  cycling  in  the  fighter  environment,  but  which  may  never  appear 
in  bomber  and  cargo  application  because  of  their  slower  changes  in  temperature 
and  longer  periods  of  relative  thermal  stability.  With  Increasing  emphasis  on 
reducing  avionics  proliferation  and  promoting  standardization,  tools  for  sup¬ 
porting  such  analyses  must  be  developed. 
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4.  REQUIREMENTS  —  ANOTHER  PERSPECTIVE 


As  described  earlier,  policy  and  technical  guidance  for  the  requirement  to 
structure  and  conduct  integrated  test  programs  already  exist.  In  today's 
environment  of  greater  need  for  standardization  because  of  increasing  opera¬ 
tional  and  support  costs,  it  is  perhaps  appropriate  to  consider  the  overriding 
requirement  of  minimizing  life-cycle  costs  as  a  function  of  applying  CERT  to 
standard  avionics  during  engineering  development. 


4.1  "Most  Representative"  Conditions 

While  the  task  of  structuring  a  CERT  test  is  challenging  enough  for  the 
case  of  a  single-host  aircraft  or  platform,  it  can  take  on  additional  dimensions 
if: 

a.  The  avionics  will  be  hosted  on  multiple  platforms 

b.  Resources  are  available  for  only  one  developmental 
test  program 

c.  A  primary  consideration  is  to  minimize  life-cycle  costs. 

For  example,  consider  the  familiar  curves  shown  in  Figure  2.  If  it  is 
assumed  that  the  i  n  curve  represents  the  improvements  in  reliability  that  are 
ideally  possible  for  given  R&D  expenditures,  then  departures  from  the  ideal  ,  as 
shown,  will  result  in  a  new,  higher  total  cost  curve,  TC'.  This  situation 
occurs  when  resources  are  invested  into  an  RAD  test  program  without  an  appre¬ 
ciable  increase  in  aggregate  (i.e.,  across  all  applications)  avionics  system 
reliability.  The  least  desirable  condition  occurs  when  no  aggregate  reliabi¬ 
lity  Improvement  is  realized  for  a  given  RAD  expenditure,  and  the  total  cost 
curve  merely  shifts  vertically  toward  TC'.  Such  an  outcome  could  result  from: 

a.  Conducting  excessive  tests 

b.  Conducting  the  wrong  tests 

c.  Conducting  tests  that  are  "right"  for 
only  a  very  small  subpopulation,  but 
applying  the  indicated  fixes  to  all  units. 
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SYSTEM  COST  (dollars) 


SYSTEM  MEAN  TIME  BETWEEN  FAILURE  (Hr) 


Figure  2.  Life-Cycle  Costs 
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Figure  3.  Simplified  Temperature  Distribution 
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If  the  CERT  requirement  is  derived  from  a  higher-level  requirement  to  mini¬ 
mize  life-cycle  or  total  cost  for  a  standard  avionics  system,  then  complying 
with  DoD  Directive  5000.40  with  respect  to  operational  realism  in  test  proce¬ 
dures  and  conditions  will  require  some  analysis  of  the  avionics  system 
population. 

The  CERT  test  objective  for  standard  avionics  should  remain  relatively 
unchanged  --  to  achieve  reliability  growth  or  reliability  qual ification  under 
conditions  that  are  most  representative  of  expected  operational  conditions.  It 
seems  that  one  of  the  first  tasks  would  be  to  assess  the  homogeneity  of  the 
avionics  equipment  population  —  i.e.,  to  what  extent  would  partitions  be 
imposed  on,  say,  a  standardized  TACAN?  Would  fighter  TACANs  always  remain 
fighter  TACANs,  or  could  such  items  be  re-cycled  back  to  tankers  or  helicopters? 
Whether  or  not  the  population  is  partitioned,  how  much  support  equipment  would 
be  required  for  each  host  platform  type? 

Consider  the  distribution  and  time  rates  of  change  of  temperatures  to  which 
a  standard  black  box  might  be  exposed.  For  discussion  purposes,  part  of  the 
three-dimensional  projection  of  that  distribution  might  appear  as  shown 
simp! istical ly  in  Figure  3.  Curve  Ci  might  represent  the  CERT  conditions  for 
a  given  platform.  But  if  Curve  C  represents  the  optimal  solution  for  all  plat¬ 
forms,  then  testing  to  Curve  Cj  conditions  would  exceed  the  optimal  CERT  con¬ 
ditions  represented  by  the  dashed  projection  onto  the  plane  of  Cj.  Such  testing 
could  then  result  in  fixes  that  are  costly  and  superfluous  for  most  of  a  stan¬ 
dard  avionics  population. 

As  stated  earlier,  the  concept  of  composite  (across  platforms)  mission  pro¬ 
files  deserves  analysis.  Equally  deserving  of  analysis  are  the  potential  con¬ 
sequences  of  testing  at  environmental  extremes  to  be  encountered  only  by 
sub-populations  —  it  may  be  very  cost  Ineffective,  for  example,  to  implement 
corrective  action  on  every  unit,  or  even  on  any  unit,  based  on  results  from  a 
CERT  test  that  imposes  (severe)  conditions  to  be  seen  in  only  a  small  percentage 
of  applications.  On  the  other  hand,  for  "most  representati ve"  or  perfectly 
selected  composite  profiles  employed  in  a  CERT  TAF  test,  every  incident  occuring 
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during  test  should  be  regarded  as  representative  of  expected  field  incidents  and 
thus  a  candidate  for  population  corrective  action. 

4.2  Effects  of  Worst  Case  Environments 


As  outlined  in  Section  3.4,  the  selection  of  the  worst  case  environment  for 
the  test  profile  may  lead  to  identification  of  failure  modes  which  may  not  occur 
in  the  most  common  of  installations.  For  example,  if  a  standard  piece  of 
avionics  is  destined  for  fighter  and  bomber  applications  and  the  population  will 
be  homogeneous  so  that  units  can  be  installed  in  any  application,  spares  may  be 
reduced  because  of  pooling  of  resources  at  the  depot  and  perhaps  even  at  collo¬ 
cated  bases.  Therefore,  the  population  is  maintained  as  homogeneous  In  order  to 
realize  O&S  cost  reduction  through  lower  spares  costs.  However,  if  the  fighter 
environment,  which  is  more  severe  than  that  of  the  bomber,  is  selected  and  makes 
up  only  a  small  percentage  of  the  total  installation,  it  is  possible  that 
failure  modes  which  are  unique  to  the  fighter  may  be  identified  and  corrected 
without  causing  any  substantial  improvement  in  bomber  O&S  costs  that  result  from 
the  majority  of  the  avionics  installations.  This  could  lead  to  Increased  aggre¬ 
gate  life-cylce  cost  since  the  cost  to  incorporate  the  design  corrections  by  the 
use  of  Engineering  Change  Proposals  (ECPs)  may  not  be  recouped  through  O&S  cost 
reduction  over  the  life  of  the  system. 

Another  possible  result  of  worst  case  environment  testing  is  the  additional 
test  time  in  a  cycle  which  may  be  necessary  in  order  to  allow  the  test  chamber 
enough  time  to  reach  the  desired  temperature.  If  an  objective  of  the  test 
program  is  to  improve  reliability  in  order  to  reduce  O&S  costs,  then  it  is 
desirable  to  run  as  many  full  mission  cycles  as  possible  in  order  to  gain  maxi¬ 
mum  data  over  the  expected  environment.  However,  by  requiring  the  test  chamber 
to  reach  extreme  environments  the  test  cycle  time  must  be  lengthened  to  allow  for 
this.  The  lengthening  of  the  test  cycle  reduces  the  number  of  cycles  that  may 
be  run  in  a  specified  period  of  time  (i.e.,  day,  week,  month,  etc.).  In  a 
program  constrained  by  schedule  and  budget,  this  results  in  fewer  cycles  and 
therefore  loss  of  mission  data  in  order  to  acquire  data  on  a  mission  phase  which 
may  not  be  germane  to  the  major  portion  of  the  avionics  environment. 
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Where  an  equipment  is  considered  mission  critical,  the  above  arguments 
against  worst  case  environment  testing  must  be  tempered  by  the  role  the  equip¬ 
ment  plays  in  the  successful  completion  of  the  mission  and  where  in  the  mission 
(before  or  after  the  need)  the  extreme  environment  occurs.  If  the  equipment  is 
required  for  the  successful  completion  of  the  mission  and  must  be  able  to 
operate  through  the  worst  case  environment,  then  the  equipment  must  be  designed 
or  improved  to  allow  for  this.  However,  where  the  equipment  is  considered  cri¬ 
tical  on  a  small  percentage  of  the  total  installations  or  where  the  ability  to 
operate  successfully  through  the  worst  case  environment  is  critical  for  a  small 
number  of  installations,  then  testing  to  worst  case  environments  may  be  an  expen¬ 
sive  method  of  improving  mission  completion  success  probability  (MCSP). 

Tradeoff  studies  may  be  necessary  to  determine  whether  the  cost  to  improve  MCSP 
for  a  small  number  of  applications  and  marginal  reduction  in  O&S  costs  justi¬ 
fies  the  price  of  required  ECPs. 

An  attractive  solution  to  the  above  problem  might  be  to  partition  the 
equipment  population  so  that  design  corrections  may  be  made  to  only  those  units 
which  will  support  the  fighter  application.  This  will  result  in  increased  life- 
cycle  cost  because  of  the  loss  of  pooling  of  resources  in  terms  of  spares.  It 
may  also  impact  other  logistics  aspects  such  as  support  equipment  if  the  support 
equipment  must  be  altered  in  order  to  verify  and  repair  failure  modes  which 
occur  under  special  environmental  conditions  or  stresses. 

As  can  be  seen  from  the  above  discussion,  the  improvement  or  measurement  of 
reliability  is  not  for  its  own  purpose  but  rather  as  a  means  to  an  end,  such  as 
reduction  of  OAS  costs  or  improvement  in  MCSP.  Therefore,  when  planning  a 
reliability  test  for  a  multiple  aircraft  application,  the  test  planner  must  be 
aware  of  the  contribution  of  the  environmental  conditions  across  applications 
before  deciding  to  focus  undue  attention  on  it  in  the  test.  Without  considering 
the  entire  situation,  design  changes  may  be  incorporated  vrtiich  will  not  alter 
the  OAS  cost  curve  sufficently  to  justify  their  expense. 


5.  IMPLICATIONS  FOR  FUTURE  CERT  APPLICATIONS 


In  this  section,  the  concept  of  minimizing  aggregate  cost  is  reiterated, 
followed  by  an  example  framework  that  could  be  useful  in  selecting  the 
appropriate  test  strategy. 

5.1  Aggregate  Cost 

In  today's  environment  of  rising  O&S  cost  and  more  standardization  of 
avionics,  a  methodology  must  be  developed  for  specifying  CERT  testing  environ¬ 
ments  such  that  life-cycle  costs  will  be  minimized.  In  the  context  of  this 
paper,  life-cycle  cost  pertains  to  the  aggregate  life-cycle  costs  across  all 
platforms  targeted  to  host  the  standard  avionics.  An  implicit  assumption  is 
that  information  such  as  target  platforms,  operational  environments,  missions, 
required  avionics  quantities,  etc.,  is  available.  Some  of  that  data  is  becoming 
available  through  the  Deputy  for  Avionics  Control's  data  collection  and  avionics 
planning  activities.  With  regard  to  the  testing  Itself,  it  is  to  be  expected 
that  CERT  profiles  optimally  derived  to  minimize  aggregate  cost  across  all  host 
platforms  may  very  well  be  technically  sub-optimal  when  considered  on  a  single- 
platform  basis.  To  reiterate:  if  the  testing  profile  results  in  actions  to 
minimize  aggregate  costs  for  standard  avionics,  the  profile  is  optimal. 

5.2  Decision  Aid 

The  selection  of  the  proper  test  strategy  has  been  discussed  in  several  of 
the  preceding  sections.  A  preliminary  decision  flow  has  been  outlined  in  Figure  4 
and  is  discussed  in  the  following  paragraphs. 

The  first  decision  one  may  be  faced  with  is  whether  the  equipment  is  meant 
for  single-host  application  or  multiple-host  applications.  This  decision  will 
probably  have  already  been  made  by  a  higher  authority  prior  to  the  test  planning 
effort.  However,  the  Issues  raised  in  Section  4  should  be  kept  in  mind  such 
as  incorporating  ECPs  in  order  to  reduce  O&S  costs  or  to  Improve  MCSP  for  small 
percentages  of  the  total  Inventory.  Having  made  the  decision  concerning  can¬ 
didate  applications,  it  is  necessary  to  generate  typical  flight  profiles  that  are 
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Figure  4. 


Test  Planning  Decision  Tool 


truly  representative.  This  would  Include  consultation  with  the  using  command, 
aircraft  manufacturer,  study  of  similar  systems,  and  use  of  MIt-STD-781  guideli¬ 
nes. 


The  decision  to  simulate  the  flight  environment  or  to  test  to  a  stimulative 
flight  environment  is  largely  based  on  the  ultimate  test  objectives.  If 
reliability  measurement  is  the  prime  objective,  then  a  simulated  environment  Is 
recommended.  This  is  because  of  the  need  to  generate  k- factors  or  other  corre¬ 
lation  techniques.  Reliability  measurements  are  extremely  useful  as  a  planning 
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tool  to  estimate  logistic  requirements  and  spares  allocations.  Additionally,  it 
may  be  useful  as  a  measurement  in  a  warranty  program  or  other  form  of  equipment 
guarantee.  However,  if  the  ultimate  objective  is  reliability  improvement  then 
a  stimulative  flight  environment  is  recommended.  A  stimulative  environment  can 
accelerate  the  identification  of  failure  modes  which  can  lead  to  earlier  design 
changes  and  greater  OSS  cost  reduction  by  avoiding  costly  retrofit  expenditures. 

The  use  of  life-cycle  cost  estimating  tools  would  be  extremely  useful  in  deter¬ 
mining  whether  it  is  advisable  to  use  simulated  environments  to  estimate 
logistic  costs  {usually  when  the  OSS  cost  curve  is  flat  so  that  changes  in 
reliability  have  little  impact  on  OSS  costs)  or  stimulative  environments  to 
improve  reliability  (usually  when  the  cost  curve  is  steep  so  that  changes  in 
reliability  have  a  substantial  impact  on  OSS  costs). 

In  the  case  of  the  multiple-host  application,  the  decision  of  whether  the 
equipment  is  to  be  treated  as  a  homogeneous  population  or  as  a  partitioned  set 
will  probably  have  been  made  by  a  higher  authority.  However,  the  tradeoffs 
discussed  in  Section  4.2  should  be  addressed  in  order  to  assess  the  impact  of 
ECPs  necessary  to  perform  the  mission  and  its  potential  for  marginal  reductions 
in  OSS  costs. 

Selecting  among  worst  case,  composite,  and  most-often-used  flight  environ¬ 
ment  is  dependent  on  the  decision  to  simulate  or  to  stimulate  and  the  com¬ 
position  of  each  environment  in  terms  of  percentage  contribution  across  the 
inventory.  Where  the  decision  is  to  stimulate  test  incidents,  the  worst  case 
baseline  environment  is  usually  selected.  However,  if  there  is  a  great 
disparity  between  applications, then  the  use  of  a  composite  environment  might  be 
more  meaningful.  If  the  decision  is  to  simulate  the  environment,  the  use  of  the 
most-often-used  flight  profile  might  be  selected  in  order  to  more  accurately 
predict  reliability  for  OSS  cost  estimation.  If  the  worst  case  environment 
under  the  stimulated  decision  branch  constitutes  only  a  small  percentage  of  the 
total  inventory,  then  stimulating  the  environment  from  the  most-often-used 
baseline  may  result  in  identifying  and  accelerating  failure  modes  which  have  a 
greater  Impact  on  total  system  life-cycle  cost.  On  the  other  hand,  if  the 
environments  are  close  so  that  little  difference  exists  between  least  severe  and 
most  severe  flight  environments,  the  worst  case  environment  should  be  selected 
under  the  simulate  condition  so  that  identification  of  failure  modes  can  be  I 

accelerated  while  gaining  maximum  OSS  cost  reduction  through  design  improvement.  I 
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Ultimately,  the  test  objectlve(s)  as  discussed  In  Section  3.3  should  be 
selected.  If  reliability  prediction  is  the  prime  goal  then  the  use  of  simulated 
environments  is  recommended.  However,  if  reliability  growth,  equipment  qualifi¬ 
cation  or  accelerated  environments  are  principal  objectives  then  stimulative 
flight  profiles  are  suggested.  The  use  of  laboratory  equipment  to  simulate  or 
to  enhance  flight  environments  makes  it  possible  to  generate  a  maximum  amount  of 
test  time  in  short  periods  of  time  as  compared  to  flight  testing.  Therefore,  it 
is  possible  to  acquire  time  acceleration  of  data  regardless  of  the  decision  to 
simulate  or  to  stimulate.  Failure  mode  isolation  is  a  major  objective  of 
reliability  measurement  or  reliability  growth  and  fits  into  either  decision. 

The  environmental  profile  can  be  closely  monitored  in  the  laboratory  and  the 
occurrence  of  equipment  malfunction  may  be  associated  with  the  environmental 
conditions  under  which  they  occur. 

6.  SUMMARY 


Combined  Environment  Reliability  Testing  (CERT)  has  been  identified  as  a 
useful  tool  in  achieving  product  reliability.  The  recent  DoD  Directive  5000.40 
requires  that: 


"...  performance,  reliability  and  environmentsal  stress 
testing  shall  be  combined,  and  types  of  environmental 
stress  will  be  combined  insofar  as  practical..." 

Therefore,  the  requirements  exist  in  terms  of  combined  environment  testing,  but 
the  question  remains  on  how  best  or  proceed. 

The  decision  to  simulate  operational  conditions  or  to  stimulate  additional 
test  incidents  by  enhanced  environments  is  based  largely  on  the  primary  test 
objective.  If  the  main  test  objective  is  to  predict  field  reliability  then  the 
use  of  a  simulated  environment  is  recommended.  However,  if  the  main  test  objec¬ 
tive  is  to  encourage  maximum  reliability  growth  then  the  use  of  a  stimulative 
environment  is  recommended. 

Before  the  decision  to  simulate  the  operational  environment  or  to  stimulate 
additional  test  incidents  is  made,  a  typical  baseline  flight  profile  should  be 
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generated  so  that  test  environments  may  be  selected  in  order  to  achieve  one  or 
more  of  the  test  objectives.  In  the  case  of  equipment  destined  for  multiple- 
host  application,  the  selection  of  the  appropriate  baseline  flight  profile  is 
complicated  by  the  need  to  cover  all  of  the  major  operating  areas  of  several 
different  aircraft  classes. 

The  appropriate  test  strategy  should  be  determined  through  an  analysis  of 
effects  on  reducing  operating  and  support  ( O&S )  costs.  Guidelines  have  been 
outlined  in  this  report  with  respect  to  tradeoffs  between  potential  O&S  cost 
savings  and  the  cost  to  incorporate  design  changes.  The  need  exists  to  develop 
decision  tqol s  to  aid  the  program  manager  or  engineer  in  selecting  the  correct 
test  strategy  and  in  establishing  the  test  environments  appropriate  to  the  class 
of  equipment  and  the  specified  applications. 
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INTRODUCTION 


This  section  presents  the  detailed  response  to  the  issues  which  were 
deliberated  by  the  four  working  groups  at  the  Combined  Environment  Reli¬ 
ability  Test  (CERT)  Workshop.  The  overall  purpose  of  the  workshop  was 
to  develop  recommendations  concerning  the  use  of  CERT  in  the  acquisition 
process.  To  achieve  this  end,  the  four  working  groups  addressed  several 
issues  pertaining  to  the  specific  topics  of  CERT  Management/Cost 
Effectiveness,  Technical  Applications,  Facilities  Capabilities,  and  MIL-STD- 
781  and  MIl-GTD-810.  The  discussions  and  recommendations  on  these  issues 
are  on  the  following  working  group  reports. 
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CERT  MANAGEMENT/COST  EFFECT I VENESS/DQD  5000.40  WORKING  GROUP  SUMMARY 


1.  SUMMARY  OF  WORKING  GROUP  ACTIVITIES 


The  CERT  Management/Cost  Effectiveness/DoD  Directive  5000.40  Working  Group 
was  chaired  by  Col.  Thomas  Musson  of  the  Office  of  the  Under-Secretary  of 
Defense  with  Mr.  Robert  Hancock  of  the  Vought  Corporation  as  co-chairman. 
Working  group  participants  are  identified  in  the  list  of  attendees  (See 
Attachment  A).  The  format  followed  by  the  working  group  was  to  begin 
with  an  open  discussion  leading  to  the  definition  of  specific  issues  for 
detailed  review  by  the  group.  Five  issues  were  identified  relating  to: 

(1)  CERT  education/explanation. 

(2)  Transition  from  concept  to  application. 

(3)  Experience  to  date  with  DoD  Directive 
5000.40  testing  guidance. 

(4)  Need  for  environmental  hardware  engineers. 

(5)  Identification  of  CERT  benefits. 

These  topics  were  addressed  by  the  group  to  varying  degrees  based  on  relative 
importance  as  perceived  by  the  group.  Emphasis  was  placed  on  cost  effec¬ 
tiveness  (in  Item  5)  which  was  broken  down  into  elements  for  detailed 
assessment  by  individual  members,  followed  by  overall  group  review. 

2.  GENERAL  DISCUSSIONS 


Working  group  activities  commenced  with  a  general  discussion  of  objectives 
and  concerns  involving  all  participants.  The  intent  was  to  surface  specific 
issues  for  detailed  review  in  subsequent  discussions.  The  following 
paragraphs  summarize  the  general  discussion  in  various  areas.  The  statements 
therein  represent  opinions  and  comments  expressed  by  individual  participants  as 
opposed  to  group  concensus. 
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2.1  Need  to  Inform  Decision  Makers 


Currently  decision  makers  do  not  have  the  information  they  need  relative 
to  whether  and  how  to  implement  CERT.  Communication  to  program  managers  is 
particularly  essential.  The  current  situation  is  such  chat  program  managers 
can  be  motivated  to  hide  failures  rather  than  find  them.  They  need  to  get  the 
message  that  realistic  reliability  testing  can  keep  their  program  "out  of  the 
newspapers"  at  a  later  date  by  uncovering  problems  early.  We  need  to  legiti¬ 
mize  failures  and  motivate  the  program  manager  to  plan  tests  with  the  specific 
objective  of  uncovering  problems.  Reliability  can  no  longer  be  viewed  as  a 
wicket  to  be  passed  through;  the  emphasis  in  reliability  testing  should  be 
realism.  The  communication  problem  should  be  addressed  for  all  three 
services. 

2.2  Definition  of  CERT 


The  question  arose  concerning  how  the  working  group  should  be  defining 
CERT  for  its  purposes.  For  example,  is  any  test  that  involves  stresses,  CERT? 
CERT  might  be  better  perceived  as  a  process  for  improving  and  measuring 
reliability.  Both  the  Air  Force  Flight  Dynamics  Laboratory  and  the  Navy 
Pacific  Missile  Test  Center  are  doing  the  right  thing  in  this  regard  -  - 
that  is,  thinking  the  problem  through.  In  the  interest  of  not  becoming  a 
cult,  it  might  be  better  not  to  specifically  define  CERT.  Past  mistakes 
relative  to  over  disciplining  the  system  acquisition  process  should  not  be 
repeated.  It  would  be  better  to  refer  to  CERT  as  an  improved  reliability 
test  approach  and  identify  a  menu  of  items  for  potential  inclusion;  such  as 
Test-Analyze-Fix  (TAF),  mission  profile  testing,  etc. 

2.3  When  and  at  What  Level  of  Assembly 


Most  testing  to  date  has  been  performed  on  equipment  already  in  the 
inventory.  Can  CERT  testing  be  moved  up  to  an  earlier  phase  in  the  system 
development  cycle?  Similarly,  there  is  the  question  of  at  what  levels  of 
assembly  should  the  various  types  of  test  be  applied.  A  sound  approach  may 
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be  to  perforin  parts  screening  and  individual  stress  testing  early  to  weed  out 
material  defects,  proceed  to  TAP  testing  at  higher  assembly  levels  to  uncover 
design  deficiencies  and  quality  problems,  and  then  perform  mission  profile 
testing  for  reliability  measurement  and/or  qualification  for  the  major 
programs.  One  industry  observation  is  that  prior  programs  have  had  CERT-type 
tests  removed  at  the  last  minute  -  -  probably  due  to  funding  concerns.  There¬ 
fore,  CERT  must  be  institutionalized  in  the  system  specification/program 
documentation. 

2.4  Transition  from  Concept  to  Application 


The  need  exists  to  identify  specific  activities  which  will  help  to 
institutionalize  CERT  into  the  system  acquisition  process.  The  relative 
advantages  and  disadvantages  of  the  top-down  versus  the  bottom-up  approach 
should  be  considered.  The  bottom-up  approach  has  the  advantage  that  the  SPO 
can  get  the  CERT  requirement  into  an  RPP  much  easier  than  can  be  achieved 
through  the  Air  Force  Regulation  route.  However,  it  will  be  very  difficult  for 
the  workshop  to  achieve  this  objective  by  going  directly  to  program  managers. 

3.  WORKING  GROUP  ISSUES 


Following  the  general  discussion,  five  issues  were  identified  for  working 
group  review: 


(1)  CERT  education/explanation. 

(2)  Transition  from  concept  to  application. 

(3)  Experience  to  date  with  DoD  Directive 
5000.40  testing  guidance. 

(4)  Need  for  environmental  hardware  engineers. 

(5)  Identification  of  CERT  benefits. 

The  last  three  issues  were  defined  for  the  working  group  In  advance  and  the 
first  two  emerged  in  the  general  discussion.  The  chairman  determined  that 
Issue  5  merited  the  highest  degree  of  attention  in  light  of  its  numerous 
aspects  and  its  relevance  to  Issues  1  and  2.  The  approach  taken  was  to  break 
Issue  2  down  in  terms  of  the  types  of  reliability  tests  at  different  phases  of 
the  system  life-cycle  as  follows: 
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-  Environmental  test  methods  (MIL-STD-810B) 

-  Reliability  development  growth  test 

-  Reliability  qualification  test 

-  DT&E/  OTAE  supplemental  testing 

-  Testing  for  source  selection 

-  Production  verification  test 

-  Testing  to  identify  field  problems  (RTOK) 

-  Maintenance  support  testing 

-  Environmental  stress  screening. 

For  each  of  these  items  a  working  group  member  was  tasked  to  prepare  a  summary 
on  the  purpose  and  role  of  the  test,  advantages  relative  to  previous 
approaches,  and  recommendations. 

4.  ISSUE  REVIEW  AND  RECOMMENDATIONS 


4.1  CERT  Education/Explanation 

The  question  addressed  was  how  to  package  the  output  of  the  workshop  for 
senior  levels  in  government  and  industry.  The  concensus  recommendation  was 
tnat,  in  addition  to  the  workshop  proceedings,  the  sponsors  should  assemble  a 
quality  presentation,  approximately  45  minutes  in  length.  Various  workshop 
members  can  take  initiatives  to  assure  that  the  presentation  will  be  seen  by 
the  appropriate  organizations.  AFSC  Headquarters  and  the  Joint  Logistics 
Commanders  were  identified  as  primary  targets  for  the  briefing.  It  was  also 
recommended  that  the  presentation  be  considered  for  the  Armament  and  Avionics 
Planning  Conference,  National  Aerospace  and  Electronics  Conference  (NAECON), 
and  the  Reliability  and  Maintainability  Symposium.  Appropriate  periodicals, 
such  as  Aviation  Week  and  the  Defense  Management  Journal,  should  also  be 
informed  of  workshop  activities.  Preparation  of  a  videotape  of  the  presen¬ 
tation  was  recommended.  An  associated  recommendation  is  to  interface  with 
appropriate  educational  institutions,  including  the  Air  Force  Institute  of 
Technology  and  Defense  Systems  Management  College. 
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4.2  Transition  from  Concept  to  Application 


The  question  addressed  was  to  identify  the  most  effective  means  of  imple¬ 
menting  the  use  of  combined  environments  testing  In  acquisition  and  modifi¬ 
cation  programs  across  the  Department  of  Defense.  History  has  been  that 
this  type  of  testing  has  been  performed  on  an  exception  basis.  The  group 
reconmends  that  DoD  issue  a  letter  to  each  of  the  three  services  requesting 
identification  of  a  focal  point  and  that  DoD  acquisition  policy  directives  be 
reviewed  in  view  of  reliability  test  policy  directed  in  DoD  Directive  5000.40 
and  the  recent  Carlucci  study.  A  secondary  recommendation,  to  support  the 
bottom-up  institutionalization  approach.  Is  that  data  from  programs  success¬ 
fully  Implementing  the  combined  environments  testing  concepts  (e.g.  Omega, 

Common  Strategic  Doppler,  Sparrow  Hisslle)  be  evaluated  and  consolidated. 

4.3  DoD  Directive  5000.40  Implementation  Experience 

DoD  Directive  5000.40  was  formally  Issued  about  eleven  months  ago,  but  has 
been  around  over  three  years.  The  issue  addressed  was  to  identify  Insights 
which  have  been  learned  that  could  be  used  to  help  any  revision  of  the  direc¬ 
tive.  The  panel  chairman  addressed  this  Issue.  In  view  of  the  recent  Ini¬ 
tiatives  launched  by  Deputy  Secretary  of  Defense  Carlucci  to  revise  the  entire 
system  acquisition  process,  the  status  of  Directive  5000.40  and  other  acquisition 
directives  are  currently  uncertain.  Therefore,  It  was  not  deemed  to  be  benefi¬ 
cial  for  the  working  group  to  make  firm  recommendations  relative  to  this  Issue. 

4.4  Need  for  Environmental  Hardware  Engineers 

The  question  addressed  was  whether  a  new  technical  discipline  called 
Environmental  Hardware  Engineering  needs  to  emerge  to  support  the  Increased 
emphasis  on  environmentally  based  tests  Including:  engineering  development, 
performance  tests,  reliability,  stress  screening,  growth,  formal  environmental 
qualification  and  any  other  tests  performed  In  the  acquisition  process. 

Individuals  skilled  In  this  discipline  shall  be  able  to  use  existing  hardware 
design  techniques,  results  of  operational  analyses,  and  knowledge  of  testing 
techniques  to  design  and/or  identify  which  environmentally  based  tests  should 
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be  included  In  an  acquisition  program.  Furthermore,  they  should  be  able  to 
determine  the  appropriate  test  conditions  and  criteria.  This  subject  received 
minimal  attention  by  the  working  group  due  to  time  spent  on  other  issues  and  no 
firm  recommendations  where  developed. 

4.5  Identification  of  CERT  Benefits 


4.5.1  MIL-STD-810B  Environmental  Test  Methods 


Alan  Burkhard  of  the  Air  Force  Wright  Aeronautical  Laboratories 
( AFWAL )  presented  this  subject  to  the  panel.  The  purpose  of  MIL-STD-810B 
testing  Is  to  verify  the  ability  of  an  equipment  design  to  function  under  (and/or 
withstand)  the  most  severe  environmental  stresses  to  be  encountered  throughout 
its  entire  life-cycle.  The  role  of  the  tests  Is  to  Identify  design  defects  and 
to  determine  contractual  compliance.  The  current  testing  approach  Is  to  subject 
a  single  equipment  Item  to  a  single  environmental  test.  Typically,  no  single 
equipment  Is  exposed  to  £lj[  anticipated  environmental  stresses.  An  Improved 
approach,  based  on  CERT  concepts,  would  be  to  combine  environmental  stresses 
when  economically  feasible.  The  potential  benefits  of  this  Improved  approach 
lie  in  a  reduction  of  the  number  of  test  and  equipment  items  required  for 
testing. 


4.5.2  Reliability  Development  Growth  Testing 

Col.  Ben  Swett  of  the  Defense  Industrial  Supply  Center  (DISC)  pre¬ 
sented  this  subject  to  the  panel.  The  purpose  of  these  tests  Is  to  detect  all 
possible  design  deficiencies  as  early  as  possible,  and  at  as  low  a  level  of 
assembly  as  possible,  during  the  full-scale  engineering  development  phase.  The 
old  development  testing  approach  relative  to  reliability  growth  was  usually 
fragmented  and  often  was  not  performed  at  all.  This  resulted  in  deficiencies 
not  being  Identified  until  system-level  development  tests  or  operational  tests. 
A  recommended  Improved  approach  would  be  to  follow  environmental  stress 
screening  and  MIL-STD-810B  tests  with  a  mission  profile  Test-Anal yze-And-Fix 
(TAAF)  program  at  the  equipment  level  prior  to  the  development  test/operational 
test  at  the  system  level.  The  potential  benefits  of  this  Improved  approach  lie 
In  the  avoidance  of  subsequent  cost  and  schedule  delays  due  to  latent  defects 
and  a  reduced  risk  of  falling  qualification  tests. 
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4.5.3  Reliability  Qualification  Test 


William  Silver  of  Westlnghouse  presented  this  subject  to  the  panel. 
The  purpose  of  the  reliability  qualification  test  Is  to  measure  the  reliability 
of  a  test  article  to  demonstrate  compliance  with  contractual  reliability 
requirements.  The  old  approach  often  results  In  testing  to  an  environmental 
profile  which  was  not  representative  of  operational  conditions.  The  result.  In 
these  circumstances.  Is  a  biased  measured  reliability  which  Is  not  useful  for 
logistics  and  operational  planning  purposes.  The  recommended  Improved  approach 
Is  to  establish  the  qualification  test  procedures  In  accordance  with  the  priori¬ 
ties  established  In  M1L-STD-781C,  Appendix  B  and.  In  particular,  to  conduct  a 
TAAF  Program  preceding  a  CERT  mission  profile  qualification  test.  The  potential 
benef 1 ts  of  this  approach  lie  In  th*»  acquisition  through  the  qualification  test, 
of  a  realistic  measure  of  field  reliability  and.  If  TAAF  Is  conducted  prior  to 
the  formal  qualification.  In  a  higher  probability  of  passing  the  test. 

4.5.4  Supplementary  Testing 

Major  James  Horkovlch  of  the  Air  Force  Test  and  Evaluation  Center 
(AFTEC)  presented  this  subject  to  the  panel.  The  purpose  of  this  testing  Is  to 
supplement  flight  and  operational  test/eval uatlon  programs  with  combined 
environments  testing.  Such  supplementary  testing  Is  not  generally  utilized 
under  the  old  approach  resulting  In  fl 1 ght/flel d  test  program  delays  while  fixes 
to  test  hardware  are  Incorporated.  The  new  approach  would  use  combined  environ¬ 
ments  testing  to  Identify  problems  before  they  cause  flight  failures.  It  Is 
especially  recommended  when  there  Is  emphasis  on  currency  In  the  development 
cycle.  The  benefits  of  supplementary  testing  He  In  the  more  efficient  utili¬ 
zation  of  test  resources,  and  In  the  ability  to  obtain  more  reliability  and 
maintainability  data  within  a  set  period  of  time,  even  In  the  presence  of 
weather  and  other  flight  delays. 

4.5.5  Source  Selection 


Robert  Gates  of  The  Analytic  Sciences  Corporation  (TASC)  presented 
this  subject  to  the  panel.  The  role  of  CERT  In  support  of  the  source  selec¬ 
tion  process  Is  to  provide  evaluators  with  meaningful  reliability  data  for 
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purposes  of  comparing  competing  system  designs.  Under  the  old  approach*  the 
evaluators  had  only  reliability  predictions  (i.e.  from  MIL-HDBK-217),  or  what¬ 
ever  data  was  submitted  In  the  proposals,  on  which  to  base  their  comparisons. 

The  Improved  approach  would  be  to  perform  CERT  (at  government  or  Independent 
facilities)  on  competing  pre-production  prototypes  prior  to  the  source  selec¬ 
tion.  The  potential  benefits  of  this  approach  lie  in  the  availability  to  eva¬ 
luators  of  realistic  reliability  predictions,  thus  making  reliability  a 
meaningful  comparison  factor.  These  predictions  can  also  be  utilized  to  deve¬ 
lop  more  realistic  life-cycle  cost  comparisons  of  the  competing  designs. 

4.5.6  Production  Verification  Testing  (PVT) 

Brent  Meeker  of  the  Pacific  Missile  Test  Center  (PMTC)  presented  this 
subject  to  the  panel.  The  purpose  of  PVT  Is  to  assure  that  the  quality  of  equip 
ment  undergoing  a  long  period  of  production  Is  not  degraded  relative  to  that 
demonstrated  during  development.  The  major  PVT  element  relating  to  reliabi¬ 
lity  Is  the  Production  Reliability  Acceptance  Test  (PRAT)  called  out  In 
MIL-STD-785B  as  Task  309.  The  old  PVT  approach  was  to  base  an  accept/reject 
decision  on  a  very  short  test  of  a  small  shape  of  production  units.  The  tests 
were  often  destructive  In  natqre.  The  Improved  approach  would  be  to  use 
realistic,  and  therefore  non-destructive,  test  environments  and  to  extend  the 
test  length  so  that  MTBF  (rather  than  pass/fail)  can  be  measured.  The  test 
results  can  then  be  fed  back  Into  the  failure  analysis  and  corrective  action 
process.  The  benefits  of  a  CERT-based  PVT  lie  In  the  Improved  field  reliabi¬ 
lity  resulting  from  this  feedback  and  better  reliability  measures  for 
logistics  and  operational  planning  purposes. 

4.5.7  Field  Problems 

Alan  Burkhard  of  AFWAL  presented  this  subject  to  the  panel.  The  sub¬ 
ject  relates  to  the  use  of  a  CERT  mission  profile  test  to  Identify  Intermit¬ 
tent  or  volatile  failures  that  can  be  observed  only  when  the  equipment  Is 
subjected  to  a  particular  stress  state.  The  current  testing  approach 
generally  will  subject  the  equipment  to  only  a  single  environmental  stress, 
based  on  a  prejudgement  as  to  which  stress  Is  Important.  The  Improved 


approach,  which  has  been  utilized  by  Newark  AFS  on  an  Inertial  Measurement 
Unit,  would  use  a  mission  profile  test  to  Identify  If  the  failure  Is  Induced 
by  stresses  that  occur  during  operational  use.  The  benefits  of  this  approach 
lie  In  the  rapid  determination  as  to  whether  the  Intermittent  or  volatile 
problem  Is  environmentally  Induced,  as  opposed  to  a  result  of  software,  test 
equipment,  technical  orders,  or  personnel  errors. 

4.5.8  CERT  for  Maintenance  Support 

Maj.  Robert  Gass  of  Air  Force  Systems  Command  (AFSC)  presented  this 
subject  to  the  panel.  This  subject  relates  to  the  establishment  of  a  combined 
environments  test  facility  at  a  depot  for  testing  returned  units  and  repaired 
units  prior  to  return  to  the  field.  The  current  depot  test  approach  often 
results  In  many  units  which  retest-OK  being  returned  to  the  pipeline.  The 
Improved  approach  would  test  such  units  In  the  CET  facility  after  they  pass 
the  normal  checkout  procedures.  The  facility  could  also  be  used  after  all 
repair  activities  are  completed  as  a  screen  to  verify  the  "goodness"  of  the 
repair.  The  benefits  of  this  approach  lie  In  the  potential  savings  In 
logistics  cost  since  It  Is  less  costly  to  repair  problems  at  depot  than  In  Jthe 
field,  and  In  Improved  spares  sufficiency  by  keeping  RTOKs  out  of  the  plpe- 
1  Ine. 

4.5.9  Environmental  Stress  Screening  (ESS) 

Richard  Baker  of  Screening  Systems,  Inc.  presented  this  subject  to  the 
panel.  The  purpose  of  ESS  Is  to  uncover  latent  equipment  manufacturing 
defects.  The  old  approach  to  ESS  has  been  somewhat  Inconsistent  and 
fragmented  relative  to  environmental  stresses  and  has  been  applied  primarily 
at  the  parts  level.  The  Improved  approach  would  be  to  use  a  combined  environ¬ 
ments  test  for  ESS  and  perform  the  test  at  all  pertinent  levels  of  assembly 
(It  may  not  be  always  oractlcal  at  the  system  level).  Furthermore,  the  ESS 
testing  should  be  performed  on  both  test  and  production  hardware.  The 
benefits  of  this  Improved  approach  lie  In  Improved  quality  control,  rapid 
problem  Identification  and  correction,  and  higher  initial  reliability  for  both 
prototype  systems  for  developmental  testing  and  operational  systems. 
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Edward  C.  Thelss 
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Larry  Midolo 
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513/255-5752 

D.  Brent  Meeker 
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805/982-7846 

Leonard  M.  Galatl 

US  Army  ARRADCOM 

201/328-6718 
AV  880-6718 

Dick  Ulddifield 
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AV  744-2202 

Robert  Gass 

AFSC/CCK 

AV  858-6869 

Mike  Ewers 
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AV  785-5694 

Maj  Dick  Algire 

HQ  AFTEC/TEBA 

AV  244-9866 

William  Silver 

Westlnghouse  Electric  Corp 

301/765-2242 

Maj  James  A.  Horkovich 

HQ  AFTEC/LG 

AV  244-0218 

Allen  L.  Curtis 

Hughes  Aircraft  Co. 

213/391-0711 

Jim  Dodd 

Wyle  Laboratories 

703/892-6700 

Dick  Baker 

Screening  Systems,  Inc. 

714/855-1751 

Ed  Andress 

Scientific  Atlanta/Spectral 
Oynamics  Division 

714/168-7160 

Alan  Burkhard 

AFWAL/FIEE-WPAFB 

513/255-6078 

Col  Ben  H.  Swett 

DISC-E  • 

215/697-3201 

Robert  K.  Gates 

The  Analytic  Sciences  Corp. 

617/944-6850 
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CERT  MGT/COST 
EFFECTIVENESS/5000.40 
WORKING  GROUP 

APPROACH 

•  GENERAL  DISCUSSION 

•  DEFINITION  OF  ISSUES 

•  REVIEW  AND  RECOMMENDATIONS 
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ISSUES  DEFINED 


•  NEED  FOR  HARDWARE  ENVIRONMENTAL 
ENGINEERS 

•  IDENTIFICATION  OF  BENEFITS 

•  5000.40  IMPLEMENTATION  EXPERIENCE 

•  TRANSITION:  CONCEPT  TO  PRACTICE 

•  EDUCATION /EXPLANATION 
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Slide  Three 


BENEFITS 


ACTIVITY 

BENEFITS 

RECOMMENDATIONS 

MILSTD-810B 

METHODS 

FEWER  TEST  SPECIMENS 

COMBINE  ENVIRONMENTS 

DEV.  TESTING 

AVOID  COST  ft  SCHEDULE 
DELAYS 

SPECIFY  FOR  ALL  FSEO 

PROGRAMS 

R-QUAL 

MORE  REALISTIC 

MEASURES  OF  R 

USE  MIL-STD-781C-PR  1 

SUPPLEMENTAL 

TESTING 

OPTIMAL  USE  OF  TEST 
RESOURCES 

EMPLOY  CERT  TO  SUPPLEMENT 

FLIGHT  FIELD  TESTS 

SOURCE  SELECTION 

MAKES  R  A  MEANINGFUL 
COMPARISON  FACTOR 

CERT  FOR  COMPETING  DESIGNS 

PROD.  VERIFICATION 

TEST 

MEANINGFUL  FEEDBACK 

TO  PRODUCTION 

DIRECT  CONSIDERATION  IN 

TEMP 

FIELD  PROBLEMS 

PROBLEM  (RTOK) 
IDENTIFICATION 

CERT  TO  IDENTIFY  VOLATILE 

FAILURE 

MAINT.  SUPPORT 

FIND  PROBLEM  AT 

DEPOT,  NOT  FIELD 

CET  CAPABILITY  AT  DEPOT 

STRESS  SCREENING 

HIGHER  INITIAL  R 
(TEST  £r  OP) 

USE  ESS  FOR  ALL 

DELIVERABLE  HARDWARE 

TECHNICAL  APPLICATIONS  WORKING  GROUP  SUMMARY 


A.  ISSUE  NR.  1: 


What  are  the  potential  uses  of  CERT?  (Initial  Issue) 

What  Is  reliability  testing?  (Issue  evolved  from  Initial  issue) 

Discussion: 


A  lengthy  discussion  followed  on  what  reliability  testing  Is  and  how  It  fits 
Into  the  overall  hardware  acquisition  process.  If  the  broad  sense  of  the  term  is 
used  to  mean  that  failure  data  is  available  for  hardware  Improvement  or  charac¬ 
terization,  then  the  following  tests  may  be  considered  as  reliability  tests: 


-  MTBF  demonstration 

-  Design  verification 

-  Reliability  growth 

-  Identification  of  failure  modes 

-  Screening 

-  Design  margin  Identification 

-  Qualification 

-  Mission  profile 


-  Production  acceptance 

-  Flight  worthiness 
-Time  acceleration 

-  Platform  compatibility  work 

verl f Icatlon/recertl f Icatl on 

-  Dormant  reliability 

-  Workmanship 


In  terms  of  reliability  as  a  function.  It  should  be  considered  an  engineering 
function  as  well  as  a  statistical  function  and,  therefore.  It  Is  part  of  the 
equipment  and  should  be  part  of  the  design  effort  rather  than  relegated  to  a  spe¬ 
cific  organizational  function  or  department. 

The  contributions  of  CERT  as  part  of  the  reliability  test  process  were  dealt 
with  In  three  categories  as  described  In  MIL-STD-785B:  deficiency  disclosure, 
operational  characterization,  and  contractual  characterization.  In  the  case  of 
deficiency  disclosure,  the  critical  need  was  Identified  for  engineering  judgement 
in  applying  CERT.  There  was  strong  reaction  against  the  blanket  application  of 
CERT.  Both  mission  profile  testing  and  so-called  worst-case  testing  have  poten¬ 
tial  benefits,  however.  It  Is  felt  that  scoring  of  test  results  (e.g.,  measuring 
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reliability  parameters)  at  this  point  was  Inappropriate  In  order  to  encourage  the 
contractor  to  Identify  and  eliminate  failure  modes. 


The  potential  benefits  of  CERT  were  Identified  as: 

i 

1.  Schedule  compression/cost  savings  due  to  test  efficiency. 

2.  Detection  of  synergistic  failure  modes  (failure  modes 

detected  only  by  the  simultaneous  application  of  more 

than  one  environmental  forcing  function). 

3.  Identification  of  Intermittent  performance  under 

environmental  stress. 

4.  Reduced  manpower/setup  time  by  testing  more  than  one 

environment  at  a  time. 

The  limitations  are: 

1.  Profiling  of  environmental  conditions  to  simulate  representa¬ 

tive  missions  Is  not  appropriate  for  screening. 

2.  Scoring  Information  not  readily  available  from  reliability 

engineering  tests. 

3.  Adequate  assets  (l.e.,  test  sample  population)  may  not  be 

available. 

4.  Facilities  may  not  be  readily  available. 

5.  Possible  need  for  special  support/diagnosis  equipment. 

CERT  can  also  be  used  for  accounting  tests.  These  were  considered  as 
those  tests  used  for  determining  reliability  data  as  an  Input  for  estimates  of 
operational  readiness,  mission  success,  maintenance  manpower  cost  and  logistic 
support  cost.  In  addition,  accounting  tests  are  used  to  determine  compliance  with 
quantitative  reliability  requirements.  It  was  felt  that  combined  environment 
testing  has  been  used  for  a  long  time  (e.g.,  AGREE  testing).  It  was  felt  that  the 
primary  advantage  of  CERT  for  accounting  test  occurs  vrften  the  equipment  Is  tested 
under  expected  operational  conditions  and  profiles.  Under  these  circumstances, 
the  government  can  expect  a  greater  likelihood  of  obtaining  a  representative 
characterization  of  operational  performance.  However,  It  was  again  emphasized 


that  CERT  Is  not  a  cookbook  and  must  be  cwsed  upon  thorough  engineering  analysis 
of  realistic  expectations  regarding  operational  use  If  It  Is  to  provide  the 
Increased  accuracy  which  is  hoped  for  In  predicting  reliability. 

Recommendations: 

1.  CERT  should  not  be  applied  as  a  blanket  methodology 

2.  CERT  should  be  based  upon  thorough,  realistic  engineering  analysis 

3.  CERT  is  and  should  be  regarded  as  more  than  mission  profile  testing. 

It  Is  applicable  to  any  situation  in  which  It  can  improve  effectiveness  or 
efficiency. 

4.  The  concept  of  mission  profile  testing  must  be  expanded  to  consider  all 
phases  of  the  hardware  life-cycle.  Life-cycle  profiles  should  include 
shipping/transport  and  storage  conditions,  when  practical,  in  addition 

to  sortie  conditions.  If  the  total  life-cycle  Is  not  considered,  signifi¬ 
cant  stress  conditions  may  be  missing  from  the  test  program,  thus  Increasing 
the  likeliness  of  critical  hardware  deficiencies  remaining  undetected  until 
operational  service. 

5.  In  applying  CERT  during  production  screening,  it  Is  Inefficient  and 
therefore,  inappropriate  to  test  using  profiles  simulating  repre¬ 
sentative  operational  missions.  Significant  life-cycle  stresses  should  be 
considered  in  designing  a  screen,  but  ultimately  the  only  environmental 
conditions  required  are  those  which  disclose  defects  quickly  and  effectively, 
regardless  of  whether  or  not  they  simulate  mission  conditions. 

B.  ISSUE  NR  2: 

Should  CERT  be  used  in  lieu  of,  or  In  conjunction  with,  flight  testing? 
Discussion: 

There  was  agreement  that  CERT  should  not  be  considered  a  complete  replacement 
for  flight  testing  or  other  operational  evaluation.  However,  CERT  can  play  an 
important  role  as  a  pre-flight  test  design  and  workmanship  screen.  The  use  of  a 
short-run  CERT  as  a  ready- for- flight  measure  to  Improve  flight  test  hardware 
reliability  could  be  extremely  useful  in  excluding  flight  safety  of  the  equipment. 
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Also,  CERT  can  provide  for  more  efficient  use  of  costly  flight  test  personnel , 
facilities,  equipment  and  range  time,  by  Improving  equipment  reliability  so  that 
planned  evaluations  can  happen  when  and  in  the  amouht  of  time  planned. 

The  question  of  the  most  appropriate  level  of  assembly  for  CERT  application  was 
raised.  Discussion  centered  around  examples  and  experiences  In  which  testing  was 
blind  to  Interface  conditions.  That  Is,  deficiencies  were  not  found  because  they 
resulted  from: 

a.  Separately  tested  units  performing  in  concert  for  the  first  time 
under  environmental  extremes. 

b.  Cable  harnesses  and  wire  bundles  between  equipment  being  outside 
of  the  test  scope. 

c.  Handling  and  troubleshooting  operations  at  a  system  level. 

It  was  felt  that  the  greater  the  level  of  assembly  at  which  CERT  Is  applied,  the 
less  likely  that  critical  deficiencies  related  to  Interface  conditions  will 
remain  undetected  in  test. 

Recommendations: 

1.  The  use  of  CERT  as  a  quick  check  of  equipment  before  flight  testing  and 
operational  evaluation. 

2.  Should  be  encouraged  to  provide  flight  safety  and  more  efficient  use  of 
costly  operational  test  and  evaluation  facilities  and  resources. 

3.  The  higher  the  level  of  assembly  at  which  CERT  is  performed,  the  greater 
the  confidence  In  the  test  results  as  useful  Indicators  of  subsequent  field 
performance. 

C.  ISSUES  NR  3: 

Should  CERT  be  based  on  so-called  worst-case  environmental  conditions 
most  likely  to  occur? 

How  should  CERT  be  applied  for  multi-application  equipment? 
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Discussion: 


A  presentation  was  made  describing  the  problem  of  selecting  test  environments 
in  terms  of  potential  impact  on  operating  and  support  (O&S)  costs.  The  issue  cen¬ 
tered  around  the  consequences  for  choosing  the  most  severe  platform  environment  as 
the  baseline  in  establishing  test  environments.  Alternatively,  what  would  be  the 
consequence  of  using  composite  or  most-likely  platform  profiles? 

It  was  decided  that  the  need  to  develop  tools  to  evaluate  O&S  costs 
and  their  impact  to  specific  failure  modes  prohibits  the  use  of  platform  environ¬ 
ments  other  than  most  severe  as  a  basis  for  establishing  test  environments. 
Engineering  judgement  and  analysis  should  be  used  to  assess  the  impact  of  correc¬ 
tive  action  for  specific  failure  modes  on  reducing  total  O&S  costs  across  all 
applications. 

Recommendations: 

Most  severe  environment.  Including  the  life-cycles  of  a  multiple  platform  or 
multiple  mission  should  be  used  as  a  baseline  in  establishing  a  test  environment 
on  design  improvement  and  disclosing  deficiencies  in  order  to  evaluate  failure 
mechanisms  over  the  expected  operational  profile.  However,  engineering  judgement 
must  be  used  to  evaluate  the  incorporation  of  design  changes  in  terms  of  cost- 
effectiveness  of  O&S  cost  reduction. 

D.  ISSUE  NR.  4: 

What  differences.  If  any,  should  there  be  between  reliability  tests 
used  for  operational  characterization  and  compliance  characterization? 

Discussion: 

It  was  generally  felt  that  representative  environmental  test  profiles  should 
be  both  types  of  characterization  tests.  The  test  profiles  should  be  based  on  all 
significant  life-cycle  environments,  and  not  limited  to  sortie  conditions.  The 
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point  was  made  that  differences  In  the  way  the  equipment  user  defined  war-time 
and  peace-time  needs  might  result  In  different  environmental  profiles  being  con¬ 
sidered  representative  for  operational  versus  contractual  characterization.  It 
was  also  noted  that  long-term  storage  environments  are  usually  not  included  as 
part  of  the  test  profile  because  of  the  large  schedule  Impact  and  because  the 
characterization  of  dormant  failure  modes  is  too  Incomplete  to  allow  realistic  and 
predictable  time  compression. 

Recommendations: 

1.  Representative  environmental  test  profiles,  based  on  anticipated 
significant  life-cycle  environmental  conditions,  should  use  realistic 
stresses  in  appropriate  time  proportions. 

2.  Long-term  storage  does  not  have  to  be  part  of  life-cycle  test  profiles. 
Although  these  conditions  should  be  considered,  their  Inclusion  should  be 
based  upon  practicality,  anticipated  benefits  and  schedule  constraints. 

3.  User  needs  should  determine  which  test  profiles  are  considered  represen¬ 
tative  for  operational  versus  contractual  reliability  characterization. 
They  may  differ. 

E.  ISSUE  NR.  5: 


Can  CERT  be  used  as  a  substitute  for,  or  in  conjunction  with,  other  tests? 
Discussion: 

The  major  advantage  of  CERT  Is  the  potential  for  streamlining  the  test  pro¬ 
cess.  For  example,  the  use  of  CERT  for  reliability  Improvement  while  also 
satisfying  qualification  or  other  requirements  listed  as  a  primary  area  of  com¬ 
bining  test  objectives. 

The  example  cited  In  the  Wagner/Burkhard  paper  (page  3.6)  presented  at  this 
workshop  was  Identified  as  an  excellent  demonstration  of  this  potential  benefit. 
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The  critical  need  for  test  integration  in  the  hardware  acquisition  was  iden¬ 
tified  as  essential  to  realizing  the  full  potential  of  CERT.  Compartmentallzation 
of  program  phases  (development,  qualification,  production)  was  seen  as  an  obstacle 
to  effective  test  Integration. 

Recommendations: 

1.  CERT  should  be  considered  as  a  method  to  replace  test  time  and  save  cost 
in  terms  of  combining  test  objectives  of  different  program  phases 
(such  as  qualification,  reliability  Improvement). 

2.  Integrated  test  planning  will  permit  the  most  effective  use  of  CERT  as 
a  tool  for  hardware  reliability  Improvement  and  characterization. 


ATTACHMENTS 

The  list  of  participants  in  the  Technical  Applications  Working  Group  is 
included  In  Attachment  A.  Copies  of  the  viewgraphs  used  in  the  final  Feedback 
Session  are  Included  in  Attachment  B. 
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CERT  FOR  DEFICIENCY 
DISCLOSURE 

•  CERT  DEFINITION  MUST  BE  AN  ENGINEERING 
TASK 

•  PERCEIVED  AS  PRIMARILY  AN  AVIONICS  TEST, 
BUT  IT  DOESN'T  HAVE  TO  BE 

•  BLANKET  APPLICATION  INAPPROPRIATE 

•  FORMAL  SCORING  (OPERATIONAL 
CHARACTERIZATION)  INAPPROPRIATE 

•  REPRESENTATIVE  MISSION-PROFILE  TESTS  AND 
SO-CALLED  WORST-CASE  TESTS  BOTH  HAVE 
POTENTIAL  BENEFITS 


POTENTIAL  CERT  BENEFITS 
(DEFICIENCY  DISCLOSURE) 


•  SCHEDULE  COMPRESSION  (AND  ASSOCIATED 
COST  SAVINGS) 

•  IDENTIFICATION  OF  SYNERGISTIC  FAILURE  MODES 

•  DISCLOSURE  OF  INTERMITTENT  PERFORMANCE 
UNDER  ENVIRONMENTAL  STRESS 

•  REDUCED  FACILITY  RELATED  MANPOWER/ 

SET-UP  TIME 

•  IMPROVED  HARDWARE  RELIABILITY  FOR 
ENVIRONMENTAL  WORTHINESS  TESTS  (SUCH  AS 
SAFETY-OF-FLIGHT) 


( 


4.3.10 


CERT  LIMITATIONS 

(DEFICIENCY  DISCLOSURE) 


•  CERT  BASED  UPON  TEST  PROFILES  SIMULATING  OPERATIONAL 
MISSIONS  INAPPROPRIATE  FOR  SCREENING  (INEFFICIENT) 

•  SCORING  (OPERATIONAL  AND  CONTRACTUAL 
CHARACTERIZATION)  INFORMATION  NOT  READILY  AVAILABLE 

•  ADEQUATE  ASSETS  (i.e.,  TEST  SAMPLES)  MAY  NOT  EXIST 

•  CERT  FACILITIES  MAY  NOT  BE  READILY  AVAILABLE: 

A.  TEST  PROGRAMS  'QUEUED'  BECAUSE  ALL  TESTS  IN  SAME 
FACILITY 

B.  FACILITIES  MAY  NOT  BE  ABLE  TO  PROVIDE  SPECIFIC 
CERT  CONDITIONS 

•  POSSIBLE  NEED  FOR  SPECIAL  SUPPORT  /DIAGNOSTIC 
EQUIPMENT 


TEST  REALISM 


•  USE  MOST  SEVERE  PLATFORM  AS 
REPRESENTATIVE,  IF  IT  IS  TYPICAL 

•  USE  EXTREME  ENVIRONMENTAL  CONDITIONS  FOR 
DEFICIENCY  DISCLOSURE  AND  DESIGN 
IMPROVEMENT 

•  USE  REPRESENTATIVE  TEST  PROFILES  BASED  ON 
LIFE-CYCLE  PROFILES  FOR  OPERATIONAL  AND 
CONTRACTUAL  CHARACTERIZATION 

•  DIFFERENT  PROFILES  MAY  BE  REPRESENTATIVE 
FOR  DIFFERENT  CHARACTERIZATION  GOALS 

•  THE  HIGHER  THE  LEVEL-OF-ASSEMBLY  TESTED, 
THE  GREATER  THE  CONFIDENCE  IN  CERT  RESULTS 
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ENGINEERING 
ANALYSIS 
AND  JUDGEMENT 
CRITICAL  TO 
SUCCESSFUL  CERT! 


TWO  VIEWS  OF  CERT 


FORCING  FUNCTION  INTEGRATION 

|  DISCIPLINARY  INTEGRATION 

(COMBINED  ENVIRONMENT) 

# 

1  COMBINED 

■  • 

j  (ENVIRONMENTAL/RELIABILITY) 

(RELIABILITY  TESTS) 

1  • 

j  TESTS 

ICE*RT] 

j  [C*E/R#T1 

THIS  VIEW  IS  TRADITIONAL 

j  THIS  VIEW  MAY  HELP 

j  ,  RESOLVE  AREAS  OF 

FACILITIES  WORKING  GROUP  SUf«ARY 


A.  ISSUE: 

Would  the  availability  of  government- owned  and  operated  CERT 
facilities  for  DoD  equipment  systems  be  of  significant  benefit 
to  the  government? 


B.  DISCUSSION: 

It  was  the  general  consensus  of  the  panel  that  CERT  be  applied.  The 
discussion  centered  on  which  types  of  testing  Is  Design  Evaluation,  TAAF, 
REL  DEMO,  Screening,  etc.  CERT  should  be  used  and  at  whose  facilities  It 
should  be  performed:  government  laboratories,  Independent  testing  labora¬ 
tories  or  contractors  In-plant  facilities. 

The  definition  of  CERT  accepted  by  the  panel  and  used  as  a  basis  for 
identifying  and  applying  CERT  facilities  Is  the  "Caruso"  definition: 

"Any  laboratory  test  for  hardware  reliability  Improvement  or  charac¬ 
terization  In  which  environmental  forcing  functions  are  applied 
simultaneously." 

There  was  early  general  agreement  that  TAAF  was  primarily  a  contractor 
function  and  responsibility  though  Instances  where  It  was  not  being  per¬ 
formed  at  the  contractor's  facility  were  cited.  Strong  emphasis  was  placed 
on  the  Importance  of  stress  screening  in  the  manufacturing  process.  It  was 
accepted  by  all  that  It  was  an  Important  CERT  type  activity  In  the  manufac¬ 
turing  process. 


There  was  an  often  repeated  discussion  on  the  multiple  application  of 
common  CERT  resources  whether  they  be  government  owned  or  industry  owned. 

A  strong  point  made  by  one  of  the  industrial  participants  was  that  the 
government  should  not  edict  the  use  of  government-owned  CERT  facilities, 
especially  when  to  do  so  would  increase  the  costs  of  performing  the  CERT 
evaluations.  The  consensus  of  the  group  is  that  a  "cookbook"  approach  to 
CERT  should  not  be  applied  but  rather  decisions  as  to  whose  and  how  much 
CERT  should  be  done,  should  be  based  on  various  factors  such  as:  purpose  of 
the  particular  CERT,  the  level  of  test,  existing  facilities,  the  need  for 
objectivity  (conform  to  OoO  5000.40),  schedule  constraints,  etc. 

A  resource  deficiency  was  identified  during  the  discussions.  At  pre¬ 
sent  there  Is  no  identified  organization  and  process  within  the  government 
with  CERT  expertise  to  perform  the  function  of  evaluating  contractor  CERT 
plans  and  programs  and  to  assist  the  program  officers  in  establishing 
contractual  requirements.  It  was  felt  that  this  is  an  Issue  for  the  CERT 
Management  Panel . 

It  was  concluded  that  the  government  needed  CERT  facilities  as  well  as 
Industry.  A  distillation  of  the  Facilities  Panel's  position  is  presented 
in  the  foil wing  paragraphs. 

Factors  Impacting  the  decision  to  develop  CERT  facilities  within  the 
government  or  at  the  contractors  facility  must  be  examined  for  each  com¬ 
modity  at  various  phases  In  the  acquisition  process.  These  factors 
Include: 


I.  Purpose  of  CERT  3. 

-  Engr.  Design  4. 

-  TAAF  5. 

-  Qualification  6. 

-  Rel .  Demo  7. 

-  Screening 

-  PATE  8. 


2.  Level  of  Test  (System 
vs  black-box) 


Existing  facilities 
Objectivity  (DoD  5000.40) 

Schedule 

Desirability  of  source  comparison 
Independent  test  facility  to  support 
competitive  development 
Technical  expertise 
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CERT  facility  requirements  between  the  government  and  contractors  are 
determined  by  the  appropriate  roles  between  the  government  and  the  producer. 
These  roles  are: 

1.  Government: 

-  Develop  test  techniques 

-  Maintain  test  specification 

-  Re view/ appraise  contractor  CERT 

-  Assess  product  reliability 

-  At  DSARC  III 

-  During  production 

-  Perform  CERT  of  out-of-production  Items 

-  Assure  Impartial  evaluation  of  competing  contractors. 

2.  Contractor: 

-  Design/Produce/Evaluate  System 
C.  RECOMMENDATIONS: 

The  recommended  facility  assignments  and  requirements  to  perform  these 
roles  are  sumnarized  as  follows: 

1.  Government: 

a.  Operate  facility  to  conduct  development  of  test 

techni ques/speci f icati ons . 

b.  Maintain  sufficient  expertise  to  assess  contractor  CERT  and 

assure  technology  transfer  from  test  development  to  program 
effort. 

c.  Provide  evaluation/correlation  of  contractor  test  facilities. 

d.  Assess  product  reliability  -  from  results  of  Rel  Demo  and  PATE. 

Test  location  at  Independent  laboratory/government  facility 
in-plant. 

2.  Contractor: 

a.  Perform  CERT  screening  as  element  of  production  process. 

b.  Perform  TAAF  at  independent  laboratory /government  facility/ 

In-plant  depending  on  program. 

c.  Perform  reliability  demonstration  test  at  Independent 

test  lab/government  faclllty/ln-plant. 
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FACILITIES 

(IN-HOUSE  vs  INDUSTRY)  WORKING  GROUP 

NAME 

AFFILIATION 

PHONE 

Jim  Perkins  -  Chairman 

PMTC,  Pt.  Mugu 

805/982-7846 

Scott  Hall  -  Co-Chairman 

AFWAL/FIEE  -  WPAFB 

513-255-6078 

Maj  Joe  Insley 

ASD/AXA  -  WPAFB 

513/255-5385 

Bill  Galbraith 

AFALD/PT  (AFLC)  -  WPAFB 

513/255-4177 

Richard  Gilfoy 

Acton  Environmental 

617/263-2933 

Russ  Levin 

Acton  Environmental 

617/263-2933 

Rudy  Volin 

Naval  Research  Lab 

202/767-3306 
AV  297-3306 

Marshal 1  Arney 

Wyle  Laboratories 

205/837-4411 

J.  Nelson  Talt 

Naval  Air  Development 

215/441-2721 

A.  J.  Karl  sen 

ARRADCOM  -  Dover,  NJ 

201/328-3585 

James  Riley 

Technology/  Scientific  Services 

513/426-2405 

John  Sheppard 

Boeing  Aerospace  -  Seattle 

206/773-2098 

Jack  Eagan 

Viking  Laboratories 

415/929-5500 

Richard  Freeman 

AFWAL/AARF-3  -  WPAFB 

513/255-5987 

513/255-3050 

Daniel  Douglas 

The  Analytic  Sciences  Corp. 

617/944-6850 
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MIL-STD-781 /APPENDIX  B  AND  MIL-STD-810  WORKING  GROUP  SUMMARY 


A.  INTRODUCTION 


The  MIL-STD-781/MIL-STD-810  Working  Group,  consisting  of  14  attendees 
(8  government/6  Industry )*,  met  to  discuss  the  differences  and  similarities 
between  MIL-STD-781 /Appendix  B  and  MIL-STD-810.  Mr.  William  Wallace  pre¬ 
sented  an  overview  of  the  new  MIL-STD-781D  draft  presently  in  a  coor¬ 
dination  cycle.  Mr.  David  Earls  presented  the  MIL-STD-810D  proposed 
draft  material  which  is  in  final  preparation  stages  for  formal  coordination 
by  1  September  1981 . 

Basically,  differences  In  the  two  standards  were  summarized  as  follows: 

1.  MIL-STD-781 /Appendix  B  tests  consist  of  up  to  4  environments, 
applied  to  electronics  equipment  in  relatively  long  duration  tests  to  obtain 
reliability  (MTBF)  Information.  The  environmental  conditions  In  MIL-STD- 

781  represent  those  levels  which  are  found  for  the  majority  of  the  time  during 
the  life-cycle  (not  extreme  conditions). 

2.  MIL-STD-810  contains  approximately  20  environments,  usually  applied  for 
short  time  durations,  and  represent  higher  level  values  associated  with  par¬ 
ticular  portions  of  the  life-cycle  profile.  MIL-STD-810  tests  apply  to  a  wide 
variety  of  equipment  categories,  much  broader  in  scope  than  electronics. 
MIL-STD-810  tests  are  conducted  to  determine  if  equipment  will  function  and 
withstand  severe  environmental  conditions  to  be  encountered  in  service. 

A  major  similarity  in  the  two  standards  includes  the  fact  that  each  standard 
requires  life-cycle  profiles  associated  with  the  equipment  to  be  tested,  along 
with  the  environmental  conditions  for  all  phases  of  the  profile. 

B.  SUMMARY  OF  ISSUES 


After  discussion  of  the  above  points  of  similarity  and  diversity,  the 
working  group  proposed  the  following  4  issues  for  further  consideration. 


*  See  Attachment  A  for  list  of  attendees. 


The  recommendations  should  be  taken  In  the  context  that  they  were  developed 
to  present  a  basis  for  resolution  to  be  considered  further.  They  should  be 
evaluated  in  a  more  deliberate  manner,  considering  more  deeply  the  ramifi¬ 
cations  of  the  work  effort  involved  and  payoff  expected.  They  are  not 
necessarily  a  consensus,  but  are  presented  for  further  detailed  investigation. 


1.  ISSUE  NR.  1 


Life-Cycle  Environmental  Profiles. 

Discussion: 

There  is  a  need  to  provide  the  methodology  and  techniques  to  develop 
and  construct  life-cycle  environmental  profiles.  Such  profiles  are  needed 
to  include  all  phases  of  the  life-cycle,  which  includes  mission,  storage, 
transportation,  etc.  As  recognized  in  MIL-STD-810  and  MIL-STD-781 ,  such 
profiles  will  differ  between  weapon  platforms  as  well  as  applications  on 
each  platform.  However,  the  basic  technique,  concept  and  necessary  con¬ 
siderations  required  to  develop  such  profiles  are  constant  (standard). 

Given  the  construction  of  a  life-cycle  profile,  design  requirements  as 
well  as  test  plans  can  be  derived  using  appropriate  portions,  environmental 
conditions  and  limits  of  the  profile.  The  environmental  stress  values 
(levels,  times,  rates  of  change,  etc.)  associated  with  the  various  portions 
of  the  profile  may  be  provided  from  a  data  base,  measurements,  estimates,  etc. 

In  this  context,  the  development  of  a  test  program  whose  objective  is 
to  characterize  reliability  (MIL-STD-781)  should  use  appropriate  portions 
of  the  life-cycle  profile  as  a  baseline.  MIL-STD-810  provides  environmen¬ 
tal  test  methods  and  procedures  covering  a  broad  range  of  weapon  platforms. 

It  also  incorporates  related  guidance  and  rationale,  including  development 
of  life-cycle  environmental  profiles. 
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Recommendation: 


a.  MIL-STD-781  and  MIL-STD-810  must  complement  each  other,  so  that  there 
Is  no  conflict.  There  should  be  no  overlap  or  redundancy.  As  an  Interim 

or  first  step,  it  is  recommended  that  Appendix  B  of  the  MIL-STD-781D  draft  be 
a  basis  for  developing  life-cycle  environmental  profiles  and  that  the  test 
methods  and  procedures  identified  in  MIL-STD-781  be  compatible  (not  conflict) 
with  MIL-STD-810  tests.  As  a  longer  range  action,  or  second  step,  environ¬ 
mental  life-cycle  profile  methods/techniques  and  test  methods/procedures 
should  be  in  MIL-STD-810  (deleted  from  MIL-STD-781)  or  life-cycle  profile 
requirements  should  be  in  a  separate  document. 

b.  MIL-STD-781  and  MIL-STD-810  were  established  for  different  purposes. 
The  environmental  life-cycle  profiles,  eventually  to  be  centralized  in 
MIL-STD-810,  should  include  all  environments  which  a  particular  piece  of 
hardware  will  experience  during  its  lifercycle.  MIL-STD-781  should  eva¬ 
luate  and  select  those  environments  and  levels  which  have  a  significant 
impact  on  long  term  reliability.  The  remaining  environments  and  short 
duration,  high  level  environmental  conditions  should  be  subjected  to 
MIL-STD-810  test  methods  and  procedures,  as  applicable. 

2.  ISSUE  NR.  2 


Environmental  Data  Base. 

Discussion: 

There  is  a  need  for  a  data  base  for  realistic  mission  profile  environ¬ 
mental  data  on  existing  platforms  and  platforms  being  designed  (e.g.,  esti¬ 
mated  levels).  This  would  be  a  common  data  base  for  use  with  both  781 D  and 
810D.  In  order  to  satisfy  the  objectives  of  781D  (i.e.,  measure  MTBF)  and 
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8100  (i.e.,  verify  design  adequacy  for  functional  and  endurance 
considerations),  data  should  include  average,  extreme  (e.g.,  95th 
percentile)  and  accelerated  test  level  algorithms.  This  data  would  be 
put  into  a  document  and  updated  as  new  platforms  come  into  existence. 

The  advantages  of  this  document  are: 

a.  Such  a  document  would  tend  to  ensure  that  realistic  mission 
profiles  would  be  used  (which  is  desired  by  the  military)  in 
preference  to  "Appendix  B"  type  data. 

b.  Quotes  from  manufacturers  for  testing  would  be  for  the  same 
tests. 

c.  Common  data  would  be  readily  available  to  procuring  agencies 
and  manufacturers. 

d.  Common  design  criteria  would  be  assured  for  the  anticipated 
application  during  the  proposal,  and  design  and  development 
stages. 

This  document  would  serve  a  role  similar  to  that  of  MIL-HDBK-5 
"Fatigue  Strength  of  Aircraft  Structural  Materials". 

Recommendation: 

Appropriate  military  agencies  should  generate  and  maintain  a  readily 
available  document  which  contains  a  common  environmental  data  base  of 
existing  platforms  for  use  with  MIL-STD-781D  and  MIL-STD-810D. 

3.  ISSUE  NR.  3 


Establish  Minimun  Vibration  Level  for  Retaining  Test  Effectiveness 
and  Increasing  Efficiency  for  Long  Duration  Testing  (Time  Sharing 
Vibration  Facilities). 
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Discussion: 


There  is  a  need  to  establish  a  lower  threshold  cutoff  level  of  vibra¬ 
tion,  where  there  Is  no  significant  impact  on  the  hardware  life,  thus 
allowing  high  cost  vibration  test  equipment  to  be  shut  down  or  time-shared 
with  other  temperature  chambers  to  Increase  overall  cost  effectiveness. 
Alternative  techniques  may  be  developed  to  accelerate  only  the  low  level 
vibration  to  cut  down  vibration  test  time.  The  MIL-STD-781C  version 
allowed  this  concept,  whereas  the  M1L-STD-781D  draft  requires  a  minimum  vibra¬ 
tion  level  to  be  continuously  applied  during  periods  when  vibration  levels  are 
actually  lower  than  the  minimum. 

Recommendation: 

Delete  the  requirement  to  vibrate  during  the  test  time  that  the 
mission  profile  vibration  level  is  equal  to  or  less  than  0.001  g2/Hz. 

4*  ISSUE  NR.  4 

Research  and  Development  Plan  for  M1L-STD-781  and  M1L-STD-810 
Discussion: 

When  M1L-STD-781  and  MIL-STD-810  are  undergoing  the  revision  process, 
it  is  common  to  observe  gaps  in  needed  data  or  technical  knowledge.  It  is 
recognized  that  more  research  and  development  effort  is  needed  to  provide 
the  test  capability  desired  and  required  for  effective  advancement  in  the 
standard  requirements.  Examples  are:  lack  of  specific  platform  data;  need 
for  accelerated  test  techniques,  chamber  simulation  techniques,  failure 
distribution  prediction,  intrinsically  high  MTBF  equipment  test  techniques; 
and  mechanical  equipment  reliability  test  techniques. 

Recommendation: 

Establish  a  five-year  research  and  development  plan  to  obtain  and 
evaluate  improved  methods,  procedures,  techniques;  gather  and  evaluate  pro¬ 
file  and  platform  data,  etc. 
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MI L-STD-781 /APPENDIX  B  A  HIL-STD-810  WORKING  GROUP 


NAME 

AFFILIATION 

PHONE 

David  Earls  -  Co-Chairman 

AFWAL/FIE-WPAFB 

513/255-6078 
AV  885-6078 

Wm  Wallace  -  Co-Chairman 

Naval  Electronic  Sys  Command 

202/692-7365 

Carmine  Colaluca 

Fairchild  Industries 

301/428-6361 

Rick  Hastings 

Texas  Instruments 

214/995-5107 

Ronald  Lambert 

General  Electric 

315/797-1000 
X  5050 

Charles  Wright 

Naval  Weapons  Center 

714/939-3468 
AV  437-3468 

Will iam  Barnes 

Warner  Robins  ALC,  GA 

912/926-5091 
AV  468-5091 

Joe  Stanley 

Warner  Robins  ALC,  GA 

912/926-2511 

Bob  Bowles 

AF  Armament  Div.  Eglin 

904/882-8652 
AV  872-8652 

Karl  Knoble 

Vought  Corporation 

214/266-5245 

Kurt  Greene 

DoD  Mat* Is  Specs  A  Stds  Office 

703/756-2343 
AV  289-2343 

Dennis  McGi vern 

Honeywell  Electro-Optics  Opts 

617/862-6222 

John  Wafford 

ASD/ENFSL-WPAFB 

513/255-4672 

Stephen  Dizek 

The  Analytic  Sciences  Corp. 

513/878-9400 

( 
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ATTENDANCE  LIST 


DOO  CERT  WORKSHOP 


2-4  June  1981 
Presidential  Hotel 
Atlanta,  Georgia 


Major  Richard  Alglre 

Chief,  Avionics  Electronics  Br. 

AFTEC/TEBA 

Klrtland  AFB,  NM  87111 
505/844-9866 

Phillip  H.  Ambs 
Member,  Technical  Staff 
The  Analytic  Sciences  Corp. 

1  Herald  Square 
Fairborn,  OH  45324 
513/878-9400 

Edwin  A.  Andress 

Mgr,  Instrument  Applications 

Spectral  Dynamics  Corporation 

Scientific  Atlanta 

P.  0.  Box  671 

San  Diego,  CA  92112 

714/268-7160 

Marshall  M.  Arney 

Mgr,  Facility  Program  Development 

Wyle  Laboratories 

P.  0.  Box  1008 

Huntsville,  AL  35807 

205/837-4411 

Richard  L.  Baker 
President 

Screening  Systems,  Inc. 

22642  Lambert  Street  #408 
El  Toro,  CA  92630 
714/855-1751 

William  C.  Barnes 
Electrical  Engineer 
Warner-Roblns  ALC/MMIRCR 
Robins  AFB,  GA  31098 
912/926-5091 

Col .  George  N .  Botbyl 
Deputy  For  Avionics  Control 
ASD-ALD/AX 
WPAFB,  OH  45433 
513/255-2734 


Bobby  G.  Bowles 
Reliability  Engineer 
Product  Assurance  Division 
US  Air  Force  Armament  Division 
Eglin  AFB,  FL  32542 
904/882-8652 

Robert  Brown 
Assistant  to  Commander 
AF  Acquisition  Logistics 
Division  (CA) 

WPAFB,  OH  45433 

Peter  Bouclin 
Environmental  Engineer 
CODE ;  3665 

Naval  Weapons  Center 
China  Lake,  CA  93555 
714/939-3468 

Alan  H.  Burkhard 

Technical  Manager 

AF  Wright  Aeronautical  Labs/FIEE 

WPAFB,  OH  45433 

513/255-6078 

E.  Keith  Burnett 

Vice  President,  Government  Systems 
Scientific  Atlanta 
Calvert  Building,  Suite  200 
6811  Kenilworth  Avenue 
Riverdale,  MD  20840 
301/779-5580 

Henry  J.  Caruso 
Senior  Engineer 

Westlnghouse  Electric  Corporation 
P.  0.  Box  746  -  M/S  504 
Baltimore,  MD  21203 
301/765-2541 

Carmine  J.  Colaluca 
Senior  Test  Engineer 
Fairchild  Space  4  Electronics  Co. 
Century  Blvd  -  M/S  A-13 
Germantown,  MD  20767 
301/428-6361 
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Allen  J.  Curtis 
Chief  Scientist 
Hughes  Aircraft  Company 
Oept  70-40-01  M/S  21/M115 
Centinela  A  Teale  Street 
Culver  City,  CA  90230 
213/391-0711  Ext  21820 

Stephen  G.  Olzek 
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